-- Logs begin at Sun 2022-08-07 00:10:30 UTC, end at Sun 2022-08-07 01:58:19 UTC. -- Aug 07 00:54:18.756471 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 07 00:54:18.766646 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Aug 7 00:54:18 2022] *** Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: nodename: dv-cinder-master-3-iscsi Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: machine: x86_64 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: clock source: unix Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: pcre jit disabled Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: detected number of CPU cores: 4 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: current working directory: / Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: detected binary path: /usr/bin/uwsgi-core Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: your processes number limit is 47680 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: your memory page size is 4096 bytes Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: detected max file descriptor number: 2048 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: lock engine: pthread robust mutexes Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: thunder lock: enabled Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 07 00:54:18.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 07 00:54:18.812704 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: Python main interpreter initialized at 0x55801e21e9c0 Aug 07 00:54:18.812704 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: python threads support enabled Aug 07 00:54:18.812704 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: your server socket listen backlog is limited to 100 connections Aug 07 00:54:18.812704 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: your mercy for graceful operations on workers is 90 seconds Aug 07 00:54:18.813084 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: mapped 403077 bytes (393 KB) for 2 cores Aug 07 00:54:18.813161 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: *** Operational MODE: preforking *** Aug 07 00:54:18.813444 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: *** uWSGI is running in multiple interpreter mode *** Aug 07 00:54:18.813444 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: spawned uWSGI master process (pid: 80164) Aug 07 00:54:18.813650 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 07 00:54:18.815010 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: spawned uWSGI worker 1 (pid: 80178, cores: 1) Aug 07 00:54:18.815010 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: spawned uWSGI worker 2 (pid: 80179, cores: 1) Aug 07 00:54:18.815010 dv-cinder-master-3-iscsi devstack@keystone.service[80164]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 07 00:54:24.460262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 07 00:54:24.460262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: from cryptography.utils import int_from_bytes Aug 07 00:54:24.461091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 07 00:54:24.461091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: from cryptography.utils import int_from_bytes Aug 07 00:54:24.461091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 07 00:54:24.461091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: from cryptography.utils import int_from_bytes Aug 07 00:54:24.461909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 07 00:54:24.461909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: from cryptography.utils import int_from_bytes Aug 07 00:54:24.512009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 07 00:54:24.512221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 07 00:54:24.512337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 07 00:54:24.512437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 07 00:54:24.512437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] command line args: [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 07 00:54:24.512601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] command line args: [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 07 00:54:24.512713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 07 00:54:24.512847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 07 00:54:24.512946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ================================================================================ {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 07 00:54:24.513080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ================================================================================ {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 07 00:54:24.513208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] admin_token = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] admin_token = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] config_dir = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] config_dir = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] config_source = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] debug = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.513961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] config_source = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] debug = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.514918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_config_append = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.515891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_dir = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_config_append = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_file = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_dir = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_file = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.516861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.517973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.518907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.519001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.519106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.519378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.519584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.519827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.519938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] publish_errors = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.520823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.521071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.521319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.521554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.26/identity {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.521739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] standard_threads = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.521739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] publish_errors = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522649 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] use_journal = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] use_json = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.522810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] use_stderr = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] use_syslog = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.523582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.523720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] standard_threads = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.523862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.523953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.524031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.524153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.524228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.524337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.524476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.524598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] use_journal = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.524707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] use_json = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.524785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.524914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] use_stderr = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.525000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.525122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] use_syslog = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.525199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.525417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 07 00:54:24.525493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.525669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.525755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.525908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.526028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.526158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.526390 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.526482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.527189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.527292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.527440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.527590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.527667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.527909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.528952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529804 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.529951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.530998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531636 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.531950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.532961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.533897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.534989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.535941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536830 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.536950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.537848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.538018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.538195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.538358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.538567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.538763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.538940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.539103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.539256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.539411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.539575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.539766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.539955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.540129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.540328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.540497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.540646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.540794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.540990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541394 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.541911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.542939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543068 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543746 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.543938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.544958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.545952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.546953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.547961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.external = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.548985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.password = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.549927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] auth.token = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.550896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.external = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.551889 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.552053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.552135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.552328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.552414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.552570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.552661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.password = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] auth.token = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.553866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.554925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.555930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.556995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.557930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.558904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.559962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.560921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.561970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562248 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.562812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.563993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.564929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.565840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.566986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.567939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.568906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.569996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.570908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.571920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.572974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.573086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.573206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.573301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.573301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.573995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.574990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.575964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.576901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.577912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.578963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.579952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.580885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.581986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.582982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] role.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.583974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] role.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.584109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.584109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.584423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.584569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.584694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.584814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.585911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.586907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] role.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] role.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.587878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.588905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.589027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.589027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.589774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.589914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.590887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.591985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.592920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.593762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.594699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.595679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.595760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.595834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.595906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.595977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.596951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.597926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.connection = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.598982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.599935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.connection = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.600919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.601870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80179) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 07 00:54:24.602496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.602964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.603200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.603449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.603749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.603933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.604164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.604427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.604739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 07 00:54:24.604918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80178) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 07 00:54:24.632968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.633230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.633526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.635216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.635550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.635685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.645081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.674221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.674462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.675718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.675951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.729785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.729993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.784951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.785276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.785914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.786184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.857925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.858057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.858144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.858279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.858370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.883850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.884888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.885023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.885114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.885243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.885341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.885462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.885679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.897928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.898260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.911445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.911676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.911906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.911995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.912918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80179) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.913055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.913055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.913254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80178) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 07 00:54:24.930395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.930592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 07 00:54:24.930941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.931125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.931264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 07 00:54:24.931392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 07 00:54:24.931534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.931676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.931773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.931854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 07 00:54:24.931924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.932118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.932286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.932417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.932665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.933444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.944442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.944735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.945012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.945189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.945317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.945432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.946013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.952926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.953537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.956029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.956344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.956637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.956745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.956926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.957833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.958921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.959106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.959377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.981990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.982449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.983519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.983810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.983980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.984299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.988506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.988796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.988959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.989096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.989228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.989489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.996361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.996660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:24.996762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:24.997013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.000877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.001150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.001241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.001451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.001536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.001738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.001821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.002984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.003133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.019645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.019937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.020290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.020595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.023978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.024566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.027283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.027585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.027861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.028158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.032025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.032602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.035297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.035794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.038463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.038767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.038984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.039983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.040166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.040477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.056803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.057099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.057327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.057414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.057625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.057705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.058505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.058780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.065148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.065879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.066722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.067006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.067457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.067741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.069872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.070164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.070459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.070594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.070761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.070912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.071048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.071172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.071467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.071752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.082590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.082883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.083165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.083329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.083447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.083627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.083733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.083911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.084014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.084198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.084360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.084502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.084780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.085060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.101276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.102012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.104288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.105553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.107777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.108514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.109413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.110089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.110917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.111215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.111502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.111617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.111874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.112174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.118004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.118581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.119642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.120186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.123238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.123532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.123775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.123858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.124075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.124158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.124391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.124676 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.132166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.132492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.132780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.132861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.133093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.133383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.139690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=80179) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.139982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.140288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.140500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=80178) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 07 00:54:25.140584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.140793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.140882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.141908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.142926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.143123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.143203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.143389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80179) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.143469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.143668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.143937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.144203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80178) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 07 00:54:25.177197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x55801e21e9c0 pid: 80179 (default app) Aug 07 00:54:25.178320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x55801e21e9c0 pid: 80178 (default app) Aug 07 00:54:25.180974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca33ac-fc3e-461f-805a-2f277510ae7d None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:25.181272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca33ac-fc3e-461f-805a-2f277510ae7d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:25.181545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca33ac-fc3e-461f-805a-2f277510ae7d None None] PATH_INFO: `/v3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:25.186447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1/1] 10.222.0.26 () {50 vars in 791 bytes} [Sun Aug 7 00:54:25 2022] GET /identity/v3/ => generated 255 bytes in 9 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:33.097399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:33.097976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:33.097976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:33.102406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1/2] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:33 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:54:33.107953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:33.108901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:33.108901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:33.178209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: INFO dbcounter [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] Registered counter for database keystone Aug 07 00:54:33.198083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG oslo_db.sqlalchemy.engines [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80179) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 07 00:54:33.205060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writer thread running {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 07 00:54:33.392040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG passlib.handlers.bcrypt [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80179) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 07 00:54:33.392408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG passlib.handlers.bcrypt [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80179) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 07 00:54:33.449055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:33.450691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:33.592430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 2/3] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:33 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 485 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 07 00:54:33.723534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:33.793520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: INFO dbcounter [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] Registered counter for database keystone Aug 07 00:54:33.815851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG oslo_db.sqlalchemy.engines [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80178) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 07 00:54:33.822266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writer thread running {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 07 00:54:33.940726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e8f7a0-f752-4638-bb15-d95b3a39fc13 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:33.944941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:33.945562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:33.945839 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:33.946108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:33.954752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:34.006809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING keystone.common.rbac_enforcer.enforcer [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 07 00:54:34.007066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebca89a-06df-41e5-b1ac-908ff5f94462 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:34.018129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 2/4] 10.222.0.26 () {58 vars in 1324 bytes} [Sun Aug 7 00:54:33 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 07 00:54:34.024370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:34.039513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610fe09-ad7a-4f61-b44e-09acd0806589 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:34.043872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:34.044994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:34.045274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:34.045547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] PATH_INFO: `/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:34.046967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] RBAC: Authorizing `identity:get_project(project_id=ae3747bdb31f45b9bc97b22fb3bfe021)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:34.097803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING keystone.common.rbac_enforcer.enforcer [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 07 00:54:34.098138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f9968f-17a0-4205-a3e9-7aa6fe3feceb admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:34.102941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 3/5] 10.222.0.26 () {56 vars in 1207 bytes} [Sun Aug 7 00:54:34 2022] GET /identity/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021 => generated 344 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:36.119844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e3ee73-8f5b-407e-b820-f44b3b7a6c2c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:36.120441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e3ee73-8f5b-407e-b820-f44b3b7a6c2c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:36.120441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e3ee73-8f5b-407e-b820-f44b3b7a6c2c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:36.124562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 3/6] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:36.130222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:36.130662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:36.130766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:36.152598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:36.154120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:36.162438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 4/7] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:36 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 07 00:54:36.185473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e3ee73-8f5b-407e-b820-f44b3b7a6c2c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:36.198539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e3ee73-8f5b-407e-b820-f44b3b7a6c2c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:36.202976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b3a255-e103-43c0-ac4b-13f2dc388474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:36.203622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3a255-e103-43c0-ac4b-13f2dc388474 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:36.204329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3a255-e103-43c0-ac4b-13f2dc388474 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:36.204623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3a255-e103-43c0-ac4b-13f2dc388474 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:36.214822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b3a255-e103-43c0-ac4b-13f2dc388474 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:36.216700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b3a255-e103-43c0-ac4b-13f2dc388474 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:36.226954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 4/8] 10.222.0.26 () {58 vars in 1324 bytes} [Sun Aug 7 00:54:36 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 00:54:36.233658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:36.243898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42310051-9eb4-4e98-a0a1-00008f16dc53 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:36.248551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859063c1-772d-406b-9517-f02217696722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:36.249223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859063c1-772d-406b-9517-f02217696722 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:36.249504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859063c1-772d-406b-9517-f02217696722 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:36.249782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859063c1-772d-406b-9517-f02217696722 admin admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:36.251261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859063c1-772d-406b-9517-f02217696722 admin admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:36.253207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859063c1-772d-406b-9517-f02217696722 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:36.254919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 5/9] 10.222.0.26 () {56 vars in 1198 bytes} [Sun Aug 7 00:54:36 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:38.382440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf5a331-e8ae-432d-ae86-06435029784a None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.382981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf5a331-e8ae-432d-ae86-06435029784a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.383086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49097df8-edde-4490-9d99-a4d4988f5fd6 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.383426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf5a331-e8ae-432d-ae86-06435029784a None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.383503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49097df8-edde-4490-9d99-a4d4988f5fd6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.383576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49097df8-edde-4490-9d99-a4d4988f5fd6 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.384377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 5/10] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:38.384862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 6/11] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:54:38.389967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.390259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.390484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.390580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.390867 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.391216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.411581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:38.412823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:38.435614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG passlib.handlers.bcrypt [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80178) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 07 00:54:38.436040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG passlib.handlers.bcrypt [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80178) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 07 00:54:38.482189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:38.483822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:38.497804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 7/12] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:38.520768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:38.546633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:38.592070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 6/13] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:38.614777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:38.635012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f031bf-9e19-4073-ad44-ea85639c09fd None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:38.675585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:38.676343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.676627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.676911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.732817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92469c55-d989-465b-b264-0f8bf657f7e1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:38.735541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:38.754155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:38.755861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:38.775282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:38.777087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.777375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.777651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.783013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:38.792483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:38.801301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:38.802404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:54:38.803498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 7/14] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:54:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:54:38.809601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:38.872962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 8/15] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:54:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 07 00:54:38.878771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:38.920686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b4a487-098b-4dae-8cd1-a02d505c6295 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:38.971663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:38.972414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:38.972701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:38.973009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:38.973709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:38.974016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:38.975534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:38.992474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 8/16] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:38.998689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:39.032017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:39.095093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbc98b3-82ea-4b3c-89d1-01afffe8fe6a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:39.118514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a479d2-dc43-4f6d-a455-380380964158 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:39.118847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:39.132569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:39.133217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:39.133501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:39.133777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:39.144282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:39.145787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:39.154688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:39.155339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:39.155626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:39.155905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:39.157477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:39.158978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:39.160685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 9/17] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:38 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:54:39.164868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:39.165796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:54:39.166608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:54:39.166608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:54:39.166608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:39.166608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 9/18] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:54:39.166837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:39.172378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:39.268227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:39.277093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98813f-8168-44ea-876b-b80e65891c86 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:39.277677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8436b7eb-7d1d-44bc-aa4e-0f80e6143d42 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:39.313094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:39.313701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:39.313977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:39.314249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:39.314671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:39.314883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:39.315180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:39.315312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:39.315596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:39.315873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:39.316602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8320374d-1c46-4247-af51-d06a8075335d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:39.317032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:39.318502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:39.320114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 10/19] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:39.326582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:39.338263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 10/20] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:54:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:39.446187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d25d12-1173-4766-ab26-8d45fb252c53 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:39.489974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:39.490615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:39.490902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:39.491303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:39.492121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.domain.id=default, user.id=a75da33429f640299cbc9644c9a49291) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:39.492460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:39.494224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f976a-ed32-4507-a31d-83aab35b5224 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:39.505676 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 11/21] 10.222.0.26 () {56 vars in 1322 bytes} [Sun Aug 7 00:54:39 2022] GET /identity/v3/role_assignments?user.id=a75da33429f640299cbc9644c9a49291&scope.domain.id=default&role.id=b0ec0093717c472da3640ec7121208d4 => generated 233 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:41.897940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0cbac5-1440-4ede-a637-219978534a6c None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:41.897940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0cbac5-1440-4ede-a637-219978534a6c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:41.897940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0cbac5-1440-4ede-a637-219978534a6c None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:41.897940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 11/22] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:41.905768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4ac10-ed0d-4337-8e55-adbcc0a38042 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:41.906773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4ac10-ed0d-4337-8e55-adbcc0a38042 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:41.907277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4ac10-ed0d-4337-8e55-adbcc0a38042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:41.928906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-47d4ac10-ed0d-4337-8e55-adbcc0a38042 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:41.930691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-47d4ac10-ed0d-4337-8e55-adbcc0a38042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:41.954979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:42.008415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e114d1-7fdc-40ea-8468-a8412b8d5bc5 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.008743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e114d1-7fdc-40ea-8468-a8412b8d5bc5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.008981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e114d1-7fdc-40ea-8468-a8412b8d5bc5 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.011390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 12/23] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:42.018814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.019470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.019989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.043254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 12/24] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:42.063327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:42.064966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:42.091203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfc0aac-87e9-41cd-bc9c-79326cb9a7d6 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.091860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfc0aac-87e9-41cd-bc9c-79326cb9a7d6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.092546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfc0aac-87e9-41cd-bc9c-79326cb9a7d6 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.094507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 13/25] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:42.100440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.101115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.101585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.132077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:42.145570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:42.148865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:42.237195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 13/26] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:42.238438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:42.309282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 14/27] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:42.311254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:42.416896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:42.436367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47835e0c-fbf2-4530-a423-9ed211cbe51a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:42.576023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c7a34c-837c-454f-a3c0-0f17195220c5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:42.617221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:42.618716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.619220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.619703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.659002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:42.659980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.660457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.660737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.665094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:42.665892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:42.677265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:42.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:42.685308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:42.686081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:42.688553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 14/28] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:54:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 451 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:42.689656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-2bae64a6-c681-4fdf-bdef-f5534d845f2c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:54:42.692501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:42.693382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:54:42.694202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:54:42.694202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:54:42.694202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:54:42.694202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 15/29] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 384 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:42.697963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55221529-8cf9-4145-9eb6-4b1de49ce367 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.698959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55221529-8cf9-4145-9eb6-4b1de49ce367 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.699315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55221529-8cf9-4145-9eb6-4b1de49ce367 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.701658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 16/30] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:42 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:42.706763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55221529-8cf9-4145-9eb6-4b1de49ce367 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:42.733051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:42.853773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b975e22e-f1db-49e6-97b2-6d1ac9c7035c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:42.879160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55221529-8cf9-4145-9eb6-4b1de49ce367 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:42.903559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d70f6d-4b98-4ea9-b664-54ce54d7634b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:42.904375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d70f6d-4b98-4ea9-b664-54ce54d7634b None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.904724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d70f6d-4b98-4ea9-b664-54ce54d7634b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.905110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d70f6d-4b98-4ea9-b664-54ce54d7634b None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.906339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d70f6d-4b98-4ea9-b664-54ce54d7634b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:42.907865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d70f6d-4b98-4ea9-b664-54ce54d7634b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:42.929014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:42.930566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:42.931220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:42.931838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:42.933249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:42.933764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:42.936183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:42.944862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:42.955165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 17/31] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:42.956784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:43.042925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 15/32] 10.222.0.26 () {60 vars in 1131 bytes} [Sun Aug 7 00:54:42 2022] POST /identity/v3/roles => generated 219 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:54:43.044814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.045332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.045812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.066239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:43.068470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:43.096058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf81c81-60b1-47c1-852c-bce5eefe4e85 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:43.192378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:43.193849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.194595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.195248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.196198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:43.196976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:43.200849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:43.224465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 18/33] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:54:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:43.225717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:43.232884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 16/34] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:43.234183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:43.269674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:43.326653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:43.401639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e75ccd3-fea4-40f0-848c-c3d314bb8e20 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:43.416001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622ecf05-909d-4c2f-aae0-284d78623672 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:43.461787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:43.462741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.463214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.464204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.465207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:43.465497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:43.466337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.466765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.467184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.467621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:43.468832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:43.470443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:43.475032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 19/35] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:43 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:43.476167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:43.504690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 17/36] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:54:43 2022] POST /identity/v3/projects => generated 301 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:54:43.506086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:43.597039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:43.663277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788da230-688e-4949-a25c-adc566fe8af4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:43.675661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2af55-d0ad-4e30-bc74-0fa5fd9e4c02 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:43.715908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fa2067-3abf-4b18-8c2c-a227f4027274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:43.718303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa2067-3abf-4b18-8c2c-a227f4027274 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.719193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa2067-3abf-4b18-8c2c-a227f4027274 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.719632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa2067-3abf-4b18-8c2c-a227f4027274 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.723406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fa2067-3abf-4b18-8c2c-a227f4027274 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:43.726271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fa2067-3abf-4b18-8c2c-a227f4027274 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:43.728982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 18/37] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:43.733976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ace1c8-d235-4859-9984-ae69daecac69 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.734385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ace1c8-d235-4859-9984-ae69daecac69 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.734670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ace1c8-d235-4859-9984-ae69daecac69 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.736321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 19/38] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:43.737171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ace1c8-d235-4859-9984-ae69daecac69 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:43.750469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:43.761780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5255f-33e4-4e20-9e14-609f8e466e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:43.762419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5255f-33e4-4e20-9e14-609f8e466e9c None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.762704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5255f-33e4-4e20-9e14-609f8e466e9c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.762979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5255f-33e4-4e20-9e14-609f8e466e9c None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.763649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf5255f-33e4-4e20-9e14-609f8e466e9c None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:43.765795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf5255f-33e4-4e20-9e14-609f8e466e9c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:43.901810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ace1c8-d235-4859-9984-ae69daecac69 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:43.902818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 20/39] 10.222.0.26 () {60 vars in 1131 bytes} [Sun Aug 7 00:54:43 2022] POST /identity/v3/roles => generated 225 bytes in 427 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:54:43.905547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.905838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.906118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.933538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:43.935178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:43.961165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46274bf-b616-4a86-8504-cd9b611ff113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:43.962952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46274bf-b616-4a86-8504-cd9b611ff113 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.963420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46274bf-b616-4a86-8504-cd9b611ff113 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.964184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46274bf-b616-4a86-8504-cd9b611ff113 None admin] PATH_INFO: `/v3/domains/default/users/a75da33429f640299cbc9644c9a49291/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.967772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46274bf-b616-4a86-8504-cd9b611ff113 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=a75da33429f640299cbc9644c9a49291, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:43.969723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46274bf-b616-4a86-8504-cd9b611ff113 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:43.973182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:43.992423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 20/40] 10.222.0.26 () {58 vars in 1361 bytes} [Sun Aug 7 00:54:43 2022] PUT /identity/v3/domains/default/users/a75da33429f640299cbc9644c9a49291/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:54:43.996494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e73af-b053-4db4-99a6-3a095c1f890d None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:43.996494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e73af-b053-4db4-99a6-3a095c1f890d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:43.996494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7e73af-b053-4db4-99a6-3a095c1f890d None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:43.997107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 21/41] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:44.016710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:44.017046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:44.017333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:44.047113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:44.048715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:44.048971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 21/42] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:44.086478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:44.164539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 22/43] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:44.189546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:44.255486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b73137e-7e50-4aac-8e0c-666c03a92537 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:44.286363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:44.309476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:44.310123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:44.310408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:44.310686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:44.311379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:44.312980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:44.326046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3efbbf5-dfa1-4bf4-a0b2-ff1df397323d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:44.332108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 22/44] 10.222.0.26 () {60 vars in 1131 bytes} [Sun Aug 7 00:54:44 2022] POST /identity/v3/roles => generated 223 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:54:44.360407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:44.375420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df516726-3aa3-4b2c-8d20-72941b2abd4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:44.376464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df516726-3aa3-4b2c-8d20-72941b2abd4f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:44.376753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df516726-3aa3-4b2c-8d20-72941b2abd4f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:44.377049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df516726-3aa3-4b2c-8d20-72941b2abd4f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:44.378312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df516726-3aa3-4b2c-8d20-72941b2abd4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:44.379870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df516726-3aa3-4b2c-8d20-72941b2abd4f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:44.381662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 23/45] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:44.388717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:44.519824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2a9112-8244-46f9-8d48-610c2dc84a19 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:44.561646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d5d66-bc86-4f14-8892-4f32d06175bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:44.562279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d5d66-bc86-4f14-8892-4f32d06175bd None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:44.562562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d5d66-bc86-4f14-8892-4f32d06175bd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:44.562851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d5d66-bc86-4f14-8892-4f32d06175bd None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:44.563511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42d5d66-bc86-4f14-8892-4f32d06175bd None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:44.565115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42d5d66-bc86-4f14-8892-4f32d06175bd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:44.595817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 23/46] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:54:44 2022] POST /identity/v3/projects => generated 312 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:54:46.149981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c4b1a-64a0-42d9-bb71-6179a3bf729b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.150527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c4b1a-64a0-42d9-bb71-6179a3bf729b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.150527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c4b1a-64a0-42d9-bb71-6179a3bf729b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.151933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 24/47] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:46.157431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.157891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.157977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.177175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:46.178645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:46.205105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:46.251986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 24/48] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:46.274917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285c4b1a-64a0-42d9-bb71-6179a3bf729b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.393320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285c4b1a-64a0-42d9-bb71-6179a3bf729b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:46.426238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:46.426791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.427070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.427342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.438796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:46.443978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:46.444862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:46.445626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 25/49] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:54:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:46.451824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.559346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b190c835-609d-4527-a26e-50aea9720ecd None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:46.593212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:46.593828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.594122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.594383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.595046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:46.595345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:46.596768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b378dbf6-f615-43c2-84d4-51d41e8c1211 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:46.609515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 25/50] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:46.610703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.681800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:46.702055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be75bc3c-f1ed-4064-9e80-b6cd7434bb9d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.702352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be75bc3c-f1ed-4064-9e80-b6cd7434bb9d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.702635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be75bc3c-f1ed-4064-9e80-b6cd7434bb9d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.703984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 26/51] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:54:46.709206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.709487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.709775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.727633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:46.728895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:46.733376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd69af60-8de5-4026-8543-a5022bf53dd8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:46.746615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:46.769812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:46.770452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.770740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.771020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.772166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:46.773661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:46.775259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 26/52] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:46 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:46.780888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.799404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 27/53] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:46.822771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.883047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3f622a-1c54-46b3-b274-66100c4ca7b7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:46.906388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:46.921663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:46.922304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.922593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.922871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.924004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:46.925510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:46.927104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 27/54] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:46.927301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbbed02-8b73-4e2d-9df5-7689311364c4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:46.933802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.963219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:46.963873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:46.964160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:46.964477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:46.965623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:46.967090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:46.968733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 28/55] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:46.974411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:46.996793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:47.036217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00451a0a-d2e9-491e-8248-7f470ca55297 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:47.070730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:47.071347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:47.071631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:47.071911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:47.072683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.domain.id=default, user.id=a75da33429f640299cbc9644c9a49291) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:47.073028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:47.074720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d70ffe-a1fc-42f5-8794-d62cabe6f24c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:47.077615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2796f786-fc7f-4ec9-b563-cbf78e35a92e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:47.088304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 28/56] 10.222.0.26 () {56 vars in 1322 bytes} [Sun Aug 7 00:54:46 2022] GET /identity/v3/role_assignments?user.id=a75da33429f640299cbc9644c9a49291&scope.domain.id=default&role.id=b0ec0093717c472da3640ec7121208d4 => generated 530 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:47.117389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73420db-a24b-4cec-a471-2ac3c333509d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:47.118028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73420db-a24b-4cec-a471-2ac3c333509d None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:47.118315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73420db-a24b-4cec-a471-2ac3c333509d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:47.118593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73420db-a24b-4cec-a471-2ac3c333509d None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:47.119264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73420db-a24b-4cec-a471-2ac3c333509d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:47.121511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73420db-a24b-4cec-a471-2ac3c333509d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:47.152150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 29/57] 10.222.0.26 () {60 vars in 1140 bytes} [Sun Aug 7 00:54:46 2022] POST /identity/v3/projects => generated 298 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:54:49.226430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf996b-1f40-46ce-b637-269ae70f78f4 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:49.226954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf996b-1f40-46ce-b637-269ae70f78f4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:49.226954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdf996b-1f40-46ce-b637-269ae70f78f4 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:49.228319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 29/58] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:49.234140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:49.234564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:49.234709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:49.253112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:49.254343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:49.321889 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 30/59] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:49.346421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf996b-1f40-46ce-b637-269ae70f78f4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:49.402050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:49.449726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdf996b-1f40-46ce-b637-269ae70f78f4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:49.484669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d6f33f-4aeb-4450-b94a-4549f9520e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:49.485222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6f33f-4aeb-4450-b94a-4549f9520e6c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:49.485509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6f33f-4aeb-4450-b94a-4549f9520e6c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:49.485786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6f33f-4aeb-4450-b94a-4549f9520e6c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:49.486906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d6f33f-4aeb-4450-b94a-4549f9520e6c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:49.488406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d6f33f-4aeb-4450-b94a-4549f9520e6c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:49.489974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 30/60] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:49.495583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:49.497810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:49.597008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56dfd36e-ffe4-4634-8733-32a3127df3ed None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:49.629828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1581599-034b-441b-91ac-6e371e5dbbe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:49.630436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1581599-034b-441b-91ac-6e371e5dbbe6 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:49.630724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1581599-034b-441b-91ac-6e371e5dbbe6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:49.631001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1581599-034b-441b-91ac-6e371e5dbbe6 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:49.631660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1581599-034b-441b-91ac-6e371e5dbbe6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:49.633152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1581599-034b-441b-91ac-6e371e5dbbe6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:49.664064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 31/61] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:54:49 2022] POST /identity/v3/users => generated 275 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:54:52.828179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a6d217-3f0c-4745-8971-ddae25d67353 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:52.829231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a6d217-3f0c-4745-8971-ddae25d67353 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:52.829710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a6d217-3f0c-4745-8971-ddae25d67353 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:52.831260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 31/62] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:52.836685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edf52dc-c32d-46c6-9bdc-203fcfbe3950 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:52.837562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edf52dc-c32d-46c6-9bdc-203fcfbe3950 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:52.838030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edf52dc-c32d-46c6-9bdc-203fcfbe3950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:52.860076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-0edf52dc-c32d-46c6-9bdc-203fcfbe3950 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:52.861668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0edf52dc-c32d-46c6-9bdc-203fcfbe3950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:52.944837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:52.963944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 32/63] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:52.999706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a6d217-3f0c-4745-8971-ddae25d67353 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.116507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d2ce98-f6d5-45d2-999d-492b89521303 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.117112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d2ce98-f6d5-45d2-999d-492b89521303 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.117624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d2ce98-f6d5-45d2-999d-492b89521303 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.119104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 33/64] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:54:53.124192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.124829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.125169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.160536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a6d217-3f0c-4745-8971-ddae25d67353 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:53.161299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:53.162520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:53.185295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:53.203344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:53.203984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.204304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.204737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.217998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:53.223820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:53.224736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:53.225511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 32/65] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:54:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:53.231790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.309293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 34/66] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:53.349674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.367369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e22c31-a283-4765-82fd-1c376544e0bb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:53.410990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:53.412176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.412639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.413223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.414420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:53.415123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:53.417295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ddd257-3b2d-4f6f-bced-f236fa43aabb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:53.434772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 33/67] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:53.436774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fabfb1b-133d-4972-9642-ab6ed5776557 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.437219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fabfb1b-133d-4972-9642-ab6ed5776557 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.437643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fabfb1b-133d-4972-9642-ab6ed5776557 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.440208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 34/68] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:53.441202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fabfb1b-133d-4972-9642-ab6ed5776557 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.493609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49fdf03-8246-4e29-85d0-d675990cffcc None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:53.497914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:53.516408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:53.536834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d9dc49-a59b-407d-8590-6edd3ba8716d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:53.537768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9dc49-a59b-407d-8590-6edd3ba8716d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.538043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9dc49-a59b-407d-8590-6edd3ba8716d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.538463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9dc49-a59b-407d-8590-6edd3ba8716d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.551626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d9dc49-a59b-407d-8590-6edd3ba8716d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-35d9dc49-a59b-407d-8590-6edd3ba8716d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:53.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:53.559034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:54:53.559830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:54:53.561786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 35/69] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:54:53.564644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.565111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.565392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.570654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fabfb1b-133d-4972-9642-ab6ed5776557 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:53.586861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:53.588090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:53.613053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d7d1fb-d03e-4faa-abc5-8b10f16550c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:53.613694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d7d1fb-d03e-4faa-abc5-8b10f16550c4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.613986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d7d1fb-d03e-4faa-abc5-8b10f16550c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.614266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d7d1fb-d03e-4faa-abc5-8b10f16550c4 None admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.615414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d7d1fb-d03e-4faa-abc5-8b10f16550c4 None admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:53.616948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d7d1fb-d03e-4faa-abc5-8b10f16550c4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:53.618793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 35/70] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:53 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:53.621652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e181f6f7-9818-48e1-9609-816fbefddf2f None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.622184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e181f6f7-9818-48e1-9609-816fbefddf2f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.622758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e181f6f7-9818-48e1-9609-816fbefddf2f None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.624794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 36/71] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:53 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:53.626012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e181f6f7-9818-48e1-9609-816fbefddf2f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.678657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 36/72] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:53.679530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.739923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e181f6f7-9818-48e1-9609-816fbefddf2f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:53.751680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:53.777229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:53.777969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.778238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.778521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.779186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:53.779488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:53.780934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3024a3-52f3-4482-8444-501722215879 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:53.787285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7047950-a738-4504-a5e0-6399c57570b1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:53.789336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 37/73] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:54:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:53.790743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2893ce-9555-45f8-8de5-6c128ed15568 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.791036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2893ce-9555-45f8-8de5-6c128ed15568 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.791313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2893ce-9555-45f8-8de5-6c128ed15568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.797088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:53.810782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0f2893ce-9555-45f8-8de5-6c128ed15568 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:53.812006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0f2893ce-9555-45f8-8de5-6c128ed15568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:53.824332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:53.824958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.825242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.825518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.881057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 38/74] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:53.882278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a66f119-9441-45ea-b519-4c3891003eb1 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:53.882541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a66f119-9441-45ea-b519-4c3891003eb1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:53.882824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a66f119-9441-45ea-b519-4c3891003eb1 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:53.884128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 39/75] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:53.888390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a66f119-9441-45ea-b519-4c3891003eb1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:53.900894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:53.902468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.001673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a66f119-9441-45ea-b519-4c3891003eb1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.003387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.008751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 37/76] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:54:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:54:54.010215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.027388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.044842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f466befb-70b8-4290-ac93-a6b3bf9bab49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.045454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f466befb-70b8-4290-ac93-a6b3bf9bab49 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.045734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f466befb-70b8-4290-ac93-a6b3bf9bab49 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.046005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f466befb-70b8-4290-ac93-a6b3bf9bab49 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.047154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f466befb-70b8-4290-ac93-a6b3bf9bab49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.048645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f466befb-70b8-4290-ac93-a6b3bf9bab49 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.050234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 40/77] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:54.055411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.055411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.055411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.072466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:54.073689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:54.122917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c27ae76-4ada-4a91-892a-322d943d3d70 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.143126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 41/78] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:54.144231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.169017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.170082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.170380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.170660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.188444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.189860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.191569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 38/79] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:54.192368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.247040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.250114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e5d1a-01ee-4b71-8772-cda4de00a0fa None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.258618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.286463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.287102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.287384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.287656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.302547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:54.302784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d713a9-aa80-498f-a909-b44d00608ddf None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:54.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:54.309167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:54.310211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 42/80] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:54.311034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.341464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.342130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.342420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.342704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.343923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.345437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.347134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 39/81] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:54.347932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.457696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b9fc5-4a41-4eca-9e80-0cb74d0d2252 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.462945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c6c66d-8bc5-41da-9b06-5771ee766c54 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.501869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.502501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.502782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.503086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.503764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.505309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.507781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.508457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.508741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.509020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.511161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.521351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:54.526724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:54.527858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:54:54.528679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 40/82] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:54.529820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.538427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 43/83] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:54:54 2022] POST /identity/v3/users => generated 289 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:54:54.539399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.562155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.659923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2426e5f2-93a1-443a-9d95-34d17658059b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.697243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5b054-6aac-455a-8889-96e681bd3a59 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.716088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.716679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.716967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.717958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.746349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.748068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.748068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.748068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.748501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:54.748706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.750062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.761886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 44/84] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:54.762572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.813335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.829623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.831780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:54.907803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:54.939843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428628d-681b-4b40-8b13-fe83f149ba6b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:54.980884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 41/85] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:54:54.981651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:54.992147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:54.992807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:54.993086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:54.993362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:54.994100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=a75da33429f640299cbc9644c9a49291) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:54.994411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:54.996072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.002740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 45/86] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/role_assignments?user.id=a75da33429f640299cbc9644c9a49291&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:55.003589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.180071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:55.187861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b35d345-4fc3-4480-a46d-d7941ff9bc4d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.213458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c87bad-bb47-4399-a351-bd70edcd3132 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.242952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.243979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.244499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.245516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.246593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:55.247113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.249080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.252054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.252905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.253193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.253472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.254703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.256158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.257995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 46/87] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:55.263068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.263260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 42/88] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:54:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:55.264667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.290331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:55.402381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2c0189-fe15-4b27-935d-917dfa47f98e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.443984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.444739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.445082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.445363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.457529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d93f3ec-0ced-44a9-9bf2-9038312d1422 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.475608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:55.495313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.495883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.496170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.496480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.497681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.499151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.501184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 47/89] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:55.501996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.534797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.536599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.589959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:55.615355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fe521a-5459-410f-821d-1bfbbcd41249 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.651723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.652516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.652872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.653224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.654463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.656174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.657563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 43/90] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:54:55.658229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.658394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 48/91] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:55.663596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.723525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:55.771825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a31e092-8943-43c1-970e-7e5e2460be2d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.796914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5397e4-d4b8-4a2c-8da1-bdb9015dad1e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:55.807427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.808139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.808508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.808854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.825128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:55.859407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:55.860466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:55.860913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:55.861494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:55.863001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=fb5de2b0ea574c64b443f39117fbe50c) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:55.863524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.866436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:55.875006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 44/92] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/role_assignments?user.id=fb5de2b0ea574c64b443f39117fbe50c&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 259 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:55.875659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:55.897877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:55.901485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:56.000577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e631d05b-b8be-4ebb-b4d5-e1478c0cfc74 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:56.024634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:56.047678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:56.048774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:56.049070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:56.049413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] PATH_INFO: `/v3/projects/5376e7cd9cd6455c9d501d8a218f735f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:56.051028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] RBAC: Authorizing `identity:get_project(project_id=5376e7cd9cd6455c9d501d8a218f735f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:56.052871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:56.055663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 45/93] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/projects/5376e7cd9cd6455c9d501d8a218f735f => generated 312 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:56.062813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:56.083523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 49/94] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:54:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:54:56.090239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:56.201660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e88ed97-f7ce-40a0-9983-c4e2dd83a69a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:56.202975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:56.243070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:56.243864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:56.244163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:56.244473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:56.245371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=5376e7cd9cd6455c9d501d8a218f735f, user.id=fb5de2b0ea574c64b443f39117fbe50c) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:56.245692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:56.247366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:56.254564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 46/95] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:54:56 2022] GET /identity/v3/role_assignments?user.id=fb5de2b0ea574c64b443f39117fbe50c&scope.project.id=5376e7cd9cd6455c9d501d8a218f735f&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:56.331477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0ea7b1-fea3-4a57-a10d-1c9c8e62c061 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:56.385822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369f91e4-24c9-4813-b249-23f9ec2eeff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:56.386486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369f91e4-24c9-4813-b249-23f9ec2eeff4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:56.386768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369f91e4-24c9-4813-b249-23f9ec2eeff4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:56.387118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369f91e4-24c9-4813-b249-23f9ec2eeff4 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:56.390385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369f91e4-24c9-4813-b249-23f9ec2eeff4 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:56.392283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369f91e4-24c9-4813-b249-23f9ec2eeff4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:56.394020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 50/96] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:54:56 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:56.400061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:56.496628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:56.544917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de14469-0669-4344-bc2b-d0d3ea202a94 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:56.582308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:56.582939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:56.583226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:56.583506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:56.584283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] RBAC: Adding query filter params (role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=fb5de2b0ea574c64b443f39117fbe50c) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:56.584600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:56.586122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a580bfa-db8e-4f7c-b7b6-3f5d413cf8b6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:56.593185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 47/97] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:54:56 2022] GET /identity/v3/role_assignments?user.id=fb5de2b0ea574c64b443f39117fbe50c&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:57.341625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ce907-3f96-45a5-998d-b21b97a9c8cc None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:57.342650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ce907-3f96-45a5-998d-b21b97a9c8cc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:57.343211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ce907-3f96-45a5-998d-b21b97a9c8cc None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:57.345122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 51/98] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:57.351663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:57.352846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:57.353430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:57.375523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:57.377329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:57.491862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 48/99] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:57.520756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74ce907-3f96-45a5-998d-b21b97a9c8cc None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:57.663741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74ce907-3f96-45a5-998d-b21b97a9c8cc None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:57.686639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:57.711659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94227919-6b0e-42be-9871-f726a4429b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:57.712714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94227919-6b0e-42be-9871-f726a4429b19 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:57.713234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94227919-6b0e-42be-9871-f726a4429b19 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:57.713754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94227919-6b0e-42be-9871-f726a4429b19 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:57.716916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94227919-6b0e-42be-9871-f726a4429b19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:57.718626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94227919-6b0e-42be-9871-f726a4429b19 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:57.720564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 52/100] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:54:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:57.727663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:57.794228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:57.861104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af2fb66-d499-43ba-bc90-639dc2110b0a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:57.907638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e84163-d731-4bbe-8a6e-7cc452f67ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:57.909076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e84163-d731-4bbe-8a6e-7cc452f67ee3 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:57.909594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e84163-d731-4bbe-8a6e-7cc452f67ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:57.912646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e84163-d731-4bbe-8a6e-7cc452f67ee3 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:57.913677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e84163-d731-4bbe-8a6e-7cc452f67ee3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:57.915453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e84163-d731-4bbe-8a6e-7cc452f67ee3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:57.955637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 49/101] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:54:57 2022] POST /identity/v3/projects => generated 302 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:54:58.011204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71434ba-a501-4e43-a021-3ce67ffaac96 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.011538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71434ba-a501-4e43-a021-3ce67ffaac96 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.011775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71434ba-a501-4e43-a021-3ce67ffaac96 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.013346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 53/102] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:58.023013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.023370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.023660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.045319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:58.046613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:58.137417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 50/103] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:58.172265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71434ba-a501-4e43-a021-3ce67ffaac96 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:58.314345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71434ba-a501-4e43-a021-3ce67ffaac96 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:58.357215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c327045b-e804-415b-96a8-c01e71f739bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:58.357966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c327045b-e804-415b-96a8-c01e71f739bc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.358147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c327045b-e804-415b-96a8-c01e71f739bc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.358422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c327045b-e804-415b-96a8-c01e71f739bc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.371332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c327045b-e804-415b-96a8-c01e71f739bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-c327045b-e804-415b-96a8-c01e71f739bc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:58.376507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:58.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:58.378770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 54/104] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:54:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:58.385229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:58.436406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc8117-e165-4efd-836b-7f17c3a852c5 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.436584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc8117-e165-4efd-836b-7f17c3a852c5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.436865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecc8117-e165-4efd-836b-7f17c3a852c5 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.438443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 55/105] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:54:58.452351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:58.460640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.460930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.461313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.499279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:58.500540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:58.579863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbedf76-a3eb-43ab-89d2-5b5bf878d12a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:58.591795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:58.638449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:58.639314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.639741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.640152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.641054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:58.641523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:58.643112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:58.656198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 51/106] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:58.663847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:58.677472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 56/107] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:54:58.757833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:58.816283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:58.821667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c1a1fd-af6c-44b9-9626-3c147e8cae2b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:58.885515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:58.887263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:58.887913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:58.889009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:58.892912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:58.895099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:58.897858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 52/108] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:58 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:58.905897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:58.967294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c5e239-17a6-481c-8715-56a918d42204 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:59.031741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df4855c-1b35-4ef5-96f6-68d6a538a1db None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:59.045055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ae2a0a-aadb-49b5-8a42-9414a67bb43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:59.045725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ae2a0a-aadb-49b5-8a42-9414a67bb43e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.046007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ae2a0a-aadb-49b5-8a42-9414a67bb43e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.046282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ae2a0a-aadb-49b5-8a42-9414a67bb43e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.073001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ae2a0a-aadb-49b5-8a42-9414a67bb43e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:59.075075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:59.076119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.076719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.077525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-44ae2a0a-aadb-49b5-8a42-9414a67bb43e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:59.078572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:59.079707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:54:59.080515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:54:59.083244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:59.085153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 57/109] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:58 2022] GET /identity/v3/roles/reader => generated 84 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:59.085269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:59.087242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 53/110] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:54:58 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:59.092290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e5d8e9-2895-4a1b-aa67-f5cc2fece278 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.092290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e5d8e9-2895-4a1b-aa67-f5cc2fece278 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.092290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e5d8e9-2895-4a1b-aa67-f5cc2fece278 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.092885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 58/111] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:54:59.093908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:59.095447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e5d8e9-2895-4a1b-aa67-f5cc2fece278 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:59.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:59.144906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:59.288856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e5d8e9-2895-4a1b-aa67-f5cc2fece278 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:59.301009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9118a34-1281-4fa1-b38f-9f17db4123bf None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:59.344182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764af59d-552c-4656-a772-4d431842e83a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:59.344994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764af59d-552c-4656-a772-4d431842e83a None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.345287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764af59d-552c-4656-a772-4d431842e83a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.345573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764af59d-552c-4656-a772-4d431842e83a None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/a75da33429f640299cbc9644c9a49291/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.347763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764af59d-552c-4656-a772-4d431842e83a None admin] RBAC: Authorizing `identity:create_grant(project_id=6fe07ee094f944898ef10a16ce01c7c3, user_id=a75da33429f640299cbc9644c9a49291, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:59.349273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764af59d-552c-4656-a772-4d431842e83a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:59.362614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:59.363975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.364487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.365059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.366223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:54:59.366814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:59.368638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8b3d30-01bc-4de4-b5e1-e833b8d56805 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:59.378832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 59/112] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:54:59 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:54:59.379831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 54/113] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:54:59 2022] PUT /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/a75da33429f640299cbc9644c9a49291/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:54:59.380676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77a7bf-0fe8-409f-b8b0-45db4bebc0ad None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.381136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77a7bf-0fe8-409f-b8b0-45db4bebc0ad None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.381658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77a7bf-0fe8-409f-b8b0-45db4bebc0ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.393717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764af59d-552c-4656-a772-4d431842e83a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:59.407001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-eb77a7bf-0fe8-409f-b8b0-45db4bebc0ad None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:59.408654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-eb77a7bf-0fe8-409f-b8b0-45db4bebc0ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:59.529286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 60/114] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:59.533626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8113be4-6524-4e0b-9812-2971bea8ceec None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.534194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8113be4-6524-4e0b-9812-2971bea8ceec None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.534652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8113be4-6524-4e0b-9812-2971bea8ceec None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.536296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 61/115] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:54:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:54:59.552545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.553338 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.554314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.572189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:59.591189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:54:59.595240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:54:59.595974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764af59d-552c-4656-a772-4d431842e83a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:59.638993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:59.664955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:59.666043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.666644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.667113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] PATH_INFO: `/v3/users/ac6f6c58b5ff4b85ab98f60d56e451b1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.688679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] RBAC: Authorizing `identity:get_user(user_id=ac6f6c58b5ff4b85ab98f60d56e451b1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:54:59.692072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:54:59.694147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 55/116] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:54:59 2022] GET /identity/v3/users/ac6f6c58b5ff4b85ab98f60d56e451b1 => generated 289 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:54:59.695440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:59.750271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 62/117] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:54:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:54:59.750271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:59.859225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c051be5-8c1b-41ad-9650-d65bb7883da1 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:54:59.901439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:54:59.902445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:54:59.902859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:54:59.903274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:54:59.916732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:54:59.922726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:54:59.923646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:54:59.924514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 56/118] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:54:59.925257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:54:59.930196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:54:59.992389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2763352b-0e1a-42f5-969a-41de423c039e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.039265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a23165c-1cce-4ac4-8823-67656d41225a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.040269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23165c-1cce-4ac4-8823-67656d41225a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.040269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23165c-1cce-4ac4-8823-67656d41225a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.041003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23165c-1cce-4ac4-8823-67656d41225a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.074151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.092385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7617ad-3e13-42df-a26f-01eb0f336a82 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.128105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23165c-1cce-4ac4-8823-67656d41225a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.129790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23165c-1cce-4ac4-8823-67656d41225a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.144067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.145563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.146206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.146751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.163095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:00.170919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:00.171868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:00.172683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:00.173883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 57/119] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:54:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:00.173883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.262473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.271643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 63/120] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:54:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:00.277153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be465a7-8fc0-4be2-a5ad-2a5f2527a56b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.278084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be465a7-8fc0-4be2-a5ad-2a5f2527a56b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.280064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be465a7-8fc0-4be2-a5ad-2a5f2527a56b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.281978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 64/121] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:00 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:00.284805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be465a7-8fc0-4be2-a5ad-2a5f2527a56b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.322863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8bdc9a-712d-472e-9c81-63b4850f60de None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.363461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.364251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.364544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.364908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.365700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:00.366014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.367160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.368466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.384420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 58/122] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:00.385375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.427798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be465a7-8fc0-4be2-a5ad-2a5f2527a56b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.479850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.480853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.481354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.482301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.483182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:00.484079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.486040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430c5f5b-bb54-4b29-908d-70ba4bf56363 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.497670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 65/123] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:00.499214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee5e419-d30b-4910-90ac-964716134fee None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.499710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee5e419-d30b-4910-90ac-964716134fee None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.500184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee5e419-d30b-4910-90ac-964716134fee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.521127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6af318-7041-412e-8e7d-b38a9138242b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.525134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-bee5e419-d30b-4910-90ac-964716134fee None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:00.529661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-bee5e419-d30b-4910-90ac-964716134fee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:00.553707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.565592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.566444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.566923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.567594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.569780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.572843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.581299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 59/124] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:00.581299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.636900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 66/125] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:00.638281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee5e419-d30b-4910-90ac-964716134fee None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.693466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.716882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a7e0d-0d22-4d6c-9931-fc81ee716368 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.754757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee5e419-d30b-4910-90ac-964716134fee None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.759114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.759962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.760964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.761386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.762810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.765130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.767016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 60/126] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:00.768171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.804986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.806130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.806755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.807157 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.808492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.810226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.812341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 67/127] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:00.812991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.868636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.921229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca42d08a-dd29-4f9e-831f-eb28a9f7e17d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.944020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67ba49-1cf5-4fcb-9ed3-d0ac2c69ce50 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:00.960742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:00.968844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.969764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.970203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.970633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.971611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] RBAC: Adding query filter params (role.id=00bbfd499c354c379adeca6c2d8ae65e, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=ac6f6c58b5ff4b85ab98f60d56e451b1) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:00.972114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:00.973738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:00.981154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 61/128] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/role_assignments?user.id=ac6f6c58b5ff4b85ab98f60d56e451b1&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=00bbfd499c354c379adeca6c2d8ae65e => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:00.982244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:00.983255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:00.983869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:00.984153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:00.984468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:00.996554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:01.001929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:01.002887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:01.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 68/129] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:01.004333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.120117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13934cd4-716d-4d4c-a12f-894ff28601a5 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.164716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.165469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.165937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.166289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.167750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.169367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.171580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 69/130] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:01 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:01.172462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.216627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440cb6b6-955f-4e50-8753-eb3bc85bf422 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.222196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:01.267608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:01.282664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.283591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.283885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.284165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] PATH_INFO: `/v3/projects/5376e7cd9cd6455c9d501d8a218f735f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.286126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] RBAC: Authorizing `identity:get_project(project_id=5376e7cd9cd6455c9d501d8a218f735f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.287903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.295063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 62/131] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:00 2022] GET /identity/v3/projects/5376e7cd9cd6455c9d501d8a218f735f => generated 312 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:01.295063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.300016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae0b940-96b5-43a1-a6df-ed7560dd032a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.341742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.342677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.343214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.343701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.344562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:01.345029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.346690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcd7d9-15f7-41e0-8792-159127268622 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.356315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 70/132] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:01.358162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c04cb-3e96-4fa2-9caa-2292604bfefc None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.358608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c04cb-3e96-4fa2-9caa-2292604bfefc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.359042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c04cb-3e96-4fa2-9caa-2292604bfefc None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.360445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 71/133] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:01.361668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624c04cb-3e96-4fa2-9caa-2292604bfefc None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.431147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcb5c55-d9cc-4dd4-b86e-e5ccbb83d053 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.469555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.470627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.471164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.471643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/fb5de2b0ea574c64b443f39117fbe50c/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.474155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] RBAC: Authorizing `identity:create_grant(project_id=6fe07ee094f944898ef10a16ce01c7c3, user_id=fb5de2b0ea574c64b443f39117fbe50c, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.475920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.495775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 63/134] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:01 2022] PUT /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/fb5de2b0ea574c64b443f39117fbe50c/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:01.498232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.504303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624c04cb-3e96-4fa2-9caa-2292604bfefc None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.504303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:01.552348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832b800f-b0d3-4a1d-80e2-1c40e121e5e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.553066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b800f-b0d3-4a1d-80e2-1c40e121e5e6 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.553348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b800f-b0d3-4a1d-80e2-1c40e121e5e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.553882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832b800f-b0d3-4a1d-80e2-1c40e121e5e6 None admin] PATH_INFO: `/v3/projects/5376e7cd9cd6455c9d501d8a218f735f/users/fb5de2b0ea574c64b443f39117fbe50c/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.556924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832b800f-b0d3-4a1d-80e2-1c40e121e5e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=5376e7cd9cd6455c9d501d8a218f735f, user_id=fb5de2b0ea574c64b443f39117fbe50c, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.558562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832b800f-b0d3-4a1d-80e2-1c40e121e5e6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.585883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 72/135] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:01 2022] PUT /identity/v3/projects/5376e7cd9cd6455c9d501d8a218f735f/users/fb5de2b0ea574c64b443f39117fbe50c/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:01.588015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.589805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.590122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.625498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:01.626956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:01.643122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:01.713178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 73/136] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:01.739474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.748016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0978657f-7826-47b5-90da-cf8e3d117691 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.790377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.792220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.792909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.793468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.795341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.797389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.799546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 64/137] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:01 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:01.808876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.851949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:01.859015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d598b932-7078-4a98-bae2-f93e7763bcb9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.897030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:01.897674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:01.897956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:01.898232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:01.899394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:01.901575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:01.903173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 74/138] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:01.908956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:01.967632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4427726d-ed36-4148-a591-a9568cbeb41c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:01.996256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:02.011264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:02.012173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:02.012658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:02.013159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:02.014649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:02.016424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:02.018471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 65/139] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:01 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:02.020138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afaf73-f714-4ed7-8ded-ad07fdb83594 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:02.033336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:02.065944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a36727-1b6e-483e-a0b8-4073de192d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:02.066921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a36727-1b6e-483e-a0b8-4073de192d25 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:02.067440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a36727-1b6e-483e-a0b8-4073de192d25 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:02.067912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a36727-1b6e-483e-a0b8-4073de192d25 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:02.068815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a36727-1b6e-483e-a0b8-4073de192d25 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:02.070559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a36727-1b6e-483e-a0b8-4073de192d25 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:02.106176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 75/140] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:01 2022] POST /identity/v3/users => generated 283 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:02.199543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4baf77-8e84-4c99-b4c4-32e435d1dc83 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:02.236631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff44d07e-e8d5-43d1-8bd1-00f87565a77e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:02.238464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff44d07e-e8d5-43d1-8bd1-00f87565a77e None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:02.238988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff44d07e-e8d5-43d1-8bd1-00f87565a77e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:02.239512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff44d07e-e8d5-43d1-8bd1-00f87565a77e None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/fb5de2b0ea574c64b443f39117fbe50c/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:02.243787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff44d07e-e8d5-43d1-8bd1-00f87565a77e None admin] RBAC: Authorizing `identity:create_grant(project_id=6fe07ee094f944898ef10a16ce01c7c3, user_id=fb5de2b0ea574c64b443f39117fbe50c, role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:02.245940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff44d07e-e8d5-43d1-8bd1-00f87565a77e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:02.273855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 66/141] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:02 2022] PUT /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/fb5de2b0ea574c64b443f39117fbe50c/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:02.612557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b6c778-2529-47da-b699-bb755d332de9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:02.613706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b6c778-2529-47da-b699-bb755d332de9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:02.614211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b6c778-2529-47da-b699-bb755d332de9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:02.616090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 76/142] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:02.624457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:02.625101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:02.625575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:02.648846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:02.651576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:02.714539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:02.741238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 67/143] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:02.766371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b6c778-2529-47da-b699-bb755d332de9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:02.790667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:02.907640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b6c778-2529-47da-b699-bb755d332de9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:02.961959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:02.963026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:02.963655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:02.964130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:02.981936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:02.987261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:02.988133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:02.989005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:02.991451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 77/144] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:02.998120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:03.119875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad52eb4-16ee-450b-a05a-b6c1e13d9688 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:03.161110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:03.162124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:03.162610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:03.163099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:03.164053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:03.164610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:03.167061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:03.181412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 68/145] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:03.188593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:03.283580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:03.314431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3e5420-9630-4571-afab-a28128a99240 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:03.351775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:03.352834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:03.353312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:03.353778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] PATH_INFO: `/v3/users/a75da33429f640299cbc9644c9a49291` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:03.355333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] RBAC: Authorizing `identity:get_user(user_id=a75da33429f640299cbc9644c9a49291)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:03.357118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:03.358961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 78/146] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:03 2022] GET /identity/v3/users/a75da33429f640299cbc9644c9a49291 => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:03.365051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:03.474874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:03.601880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d620baa9-7015-45c0-8269-3956fda026c1 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:03.651372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:03.652013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:03.652272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:03.652553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:03.737438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:03.738872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:03.795354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:03.864957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 69/147] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:03.871357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:04.021297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7699474-c051-4fe1-b1b3-08b4798c6716 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:04.062219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19851e7-9e8d-4c80-9d5f-c89ad5dfd6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:04.063335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19851e7-9e8d-4c80-9d5f-c89ad5dfd6ce None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:04.063869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19851e7-9e8d-4c80-9d5f-c89ad5dfd6ce None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:04.064386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19851e7-9e8d-4c80-9d5f-c89ad5dfd6ce None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:04.065946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19851e7-9e8d-4c80-9d5f-c89ad5dfd6ce None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:04.067723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19851e7-9e8d-4c80-9d5f-c89ad5dfd6ce None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:04.069792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 79/148] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:03 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:04.075893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:04.221296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24b66a7-ae39-485c-81f3-977dd9bfad75 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:04.261759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:04.262383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:04.262658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:04.262926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:04.263666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=a75da33429f640299cbc9644c9a49291) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:04.263967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:04.265461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c77f23c-4e3d-4abd-ad29-7cb3a13827eb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:04.286718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 70/149] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:04 2022] GET /identity/v3/role_assignments?user.id=a75da33429f640299cbc9644c9a49291&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:04.425363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f31a8-fd6d-423f-89fa-d792dd7d3eb5 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:04.426075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f31a8-fd6d-423f-89fa-d792dd7d3eb5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:04.426574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f31a8-fd6d-423f-89fa-d792dd7d3eb5 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:04.428916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 80/150] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:04.435065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:04.435358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:04.435615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:04.455636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:04.464368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:04.465603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:04.557003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 71/151] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:04.604798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f31a8-fd6d-423f-89fa-d792dd7d3eb5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:04.630026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:04.741835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f31a8-fd6d-423f-89fa-d792dd7d3eb5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:04.778981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:04.779624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:04.779901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:04.780174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:04.794120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:04.798992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:04.799869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:04.801071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 81/152] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:04.806471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:04.930628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d239e79f-dc6f-4a77-be74-d18c99d4a81e None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:04.971946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:04.972605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:04.972882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:04.973154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:04.973890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:04.974195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:04.975579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:04.986369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 72/153] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:04.988621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:04.991761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:05.082151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:05.117862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92343361-7caa-4243-ab24-e36e9601bbd8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:05.155776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:05.156445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.156724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.156999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] PATH_INFO: `/v3/users/ac6f6c58b5ff4b85ab98f60d56e451b1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.158179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] RBAC: Authorizing `identity:get_user(user_id=ac6f6c58b5ff4b85ab98f60d56e451b1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:05.159666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:05.161332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 82/154] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:04 2022] GET /identity/v3/users/ac6f6c58b5ff4b85ab98f60d56e451b1 => generated 289 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:05.167115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:05.279608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a7dada-14d9-4bc5-8f0a-fc3c641c24f0 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:05.319518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15803604-af93-433c-bef1-8175efe87f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:05.320251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15803604-af93-433c-bef1-8175efe87f8b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.320543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15803604-af93-433c-bef1-8175efe87f8b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.320820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15803604-af93-433c-bef1-8175efe87f8b None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.322108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15803604-af93-433c-bef1-8175efe87f8b None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:05.324395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15803604-af93-433c-bef1-8175efe87f8b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:05.326267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 73/155] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:05 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:05.332386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:05.458060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc278d-3a10-4a16-ac39-3b8ac5a2bcd3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:05.493723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9afa95-f5b8-4ee5-9daf-3456fec82bfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:05.494494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9afa95-f5b8-4ee5-9daf-3456fec82bfd None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.494807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9afa95-f5b8-4ee5-9daf-3456fec82bfd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.495078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9afa95-f5b8-4ee5-9daf-3456fec82bfd None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/ac6f6c58b5ff4b85ab98f60d56e451b1/roles/00bbfd499c354c379adeca6c2d8ae65e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.497119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9afa95-f5b8-4ee5-9daf-3456fec82bfd None admin] RBAC: Authorizing `identity:create_grant(project_id=6fe07ee094f944898ef10a16ce01c7c3, user_id=ac6f6c58b5ff4b85ab98f60d56e451b1, role_id=00bbfd499c354c379adeca6c2d8ae65e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:05.498578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9afa95-f5b8-4ee5-9daf-3456fec82bfd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:05.518833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 83/156] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:05 2022] PUT /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/users/ac6f6c58b5ff4b85ab98f60d56e451b1/roles/00bbfd499c354c379adeca6c2d8ae65e => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:05.575750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864782a3-6510-4ba8-900e-a24b9dc1c409 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.576106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864782a3-6510-4ba8-900e-a24b9dc1c409 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.576360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864782a3-6510-4ba8-900e-a24b9dc1c409 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.577780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 74/157] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:05.591016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dce476-c3d1-4e4d-8833-046d5208cee7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dce476-c3d1-4e4d-8833-046d5208cee7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.591569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dce476-c3d1-4e4d-8833-046d5208cee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.602136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:05.621221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-99dce476-c3d1-4e4d-8833-046d5208cee7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:05.622463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-99dce476-c3d1-4e4d-8833-046d5208cee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:05.706440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 84/158] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:05.716536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4bab2-20dc-4cfe-99e7-aca1fffb5b72 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.716834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4bab2-20dc-4cfe-99e7-aca1fffb5b72 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.717086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4bab2-20dc-4cfe-99e7-aca1fffb5b72 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.718508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 75/159] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:55:05.724396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.724702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.724945 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:05.746113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:05.747386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:05.780625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d4bab2-20dc-4cfe-99e7-aca1fffb5b72 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:05.830050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 85/160] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:05.879614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:05.888719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:05.929692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d4bab2-20dc-4cfe-99e7-aca1fffb5b72 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:05.988631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:05.993078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:05.994023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:05.994592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:05.995288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.017158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:06.030647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:06.031613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:06.032433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 76/161] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:06.040121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:06.071354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643e09e6-fa9a-4736-b7ca-30bbb637cf8a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:06.149343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:06.150051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.150458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.150808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.170232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:06.180265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:06.181236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:06.182060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:06.184771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 86/162] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:06.190641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:06.205608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fe6d7b-4994-4077-abb6-ffd6bffc9b46 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:06.235703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:06.246472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:06.246818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.247076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.247362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.248056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:06.248445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:06.249898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:06.261023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 77/163] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:06.266936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:06.389906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5891b0-496e-4ed7-932c-b143242de13b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:06.430499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0976ee26-66e0-4262-9323-9c75442d256d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:06.431173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0976ee26-66e0-4262-9323-9c75442d256d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.431463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0976ee26-66e0-4262-9323-9c75442d256d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.431743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0976ee26-66e0-4262-9323-9c75442d256d None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.433491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0976ee26-66e0-4262-9323-9c75442d256d None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:06.435076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0976ee26-66e0-4262-9323-9c75442d256d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:06.441431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 78/164] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:06 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:06.442773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ebce46-8489-4b7a-b144-e5109d1b7464 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.443179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ebce46-8489-4b7a-b144-e5109d1b7464 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.443546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ebce46-8489-4b7a-b144-e5109d1b7464 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.444872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 79/165] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:06.446188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ebce46-8489-4b7a-b144-e5109d1b7464 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:06.512696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:06.547594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2b7e8a-95f9-4e67-a71b-b811e5100dca None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:06.573117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ebce46-8489-4b7a-b144-e5109d1b7464 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:06.607509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:06.633627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:06.634253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.634534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.634806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.635560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:06.635860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:06.637280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e071d94f-6091-47ec-aa71-8e5827efea8e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:06.647064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:06.647684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.647968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.648310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.657421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 87/166] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 467 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:06.661026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:06.661314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:06.661592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:06.715239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:06.716480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:06.772550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:06.774861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:06.872136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 88/167] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:06.873103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:06.910897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:07.037919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734e466-b111-480a-99d9-9724aac601e7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:07.044346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 80/168] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 594 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:07.045361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:07.074534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:07.118713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:07.120267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.120558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.120835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.124405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:07.126066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:07.127893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 89/169] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:06 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:07.133772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:07.274396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d01ae-6629-4e65-a3d9-af91e795d0df None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:07.302346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa0c6c6-b9ea-41ac-baeb-8dff64f4edef None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:07.368995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:07.370165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.370447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.370717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] PATH_INFO: `/v3/projects/5376e7cd9cd6455c9d501d8a218f735f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.371927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] RBAC: Authorizing `identity:get_project(project_id=5376e7cd9cd6455c9d501d8a218f735f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:07.373415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:07.375235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 90/170] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:07 2022] GET /identity/v3/projects/5376e7cd9cd6455c9d501d8a218f735f => generated 312 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:07.375940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:07.401601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:07.402615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.403113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.403586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.430949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:07.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:07.437437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:07.438238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 81/171] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 393 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:07.438892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:07.475148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:07.485128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:07.532415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5958d2-dcf2-4144-b552-cb54c8b24b6b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:07.567357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f07960-6278-42f1-bc90-d33ffe6f553f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:07.567866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f07960-6278-42f1-bc90-d33ffe6f553f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.568142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f07960-6278-42f1-bc90-d33ffe6f553f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.568442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f07960-6278-42f1-bc90-d33ffe6f553f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.640639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f07960-6278-42f1-bc90-d33ffe6f553f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:07.642123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f07960-6278-42f1-bc90-d33ffe6f553f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:07.650232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37659d1b-59d7-4ff1-9b87-f40da17836fa None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:07.699813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:07.700578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.700859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.701135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.701891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=5376e7cd9cd6455c9d501d8a218f735f, user.id=fb5de2b0ea574c64b443f39117fbe50c) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:07.702202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:07.704274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:07.705028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec0f8e9-8426-4f3d-bfea-5e74d70e3242 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:07.713802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 82/172] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:07 2022] GET /identity/v3/role_assignments?user.id=fb5de2b0ea574c64b443f39117fbe50c&scope.project.id=5376e7cd9cd6455c9d501d8a218f735f&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 608 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:07.715472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57959925-b9bc-438d-8daf-6fab8683708d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.716051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57959925-b9bc-438d-8daf-6fab8683708d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.716366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57959925-b9bc-438d-8daf-6fab8683708d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.718522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 83/173] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:07.719643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57959925-b9bc-438d-8daf-6fab8683708d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:07.810617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 91/174] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:07.812095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c323553-3a86-4b55-8acd-e85dffad9ab7 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.812586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c323553-3a86-4b55-8acd-e85dffad9ab7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.813040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c323553-3a86-4b55-8acd-e85dffad9ab7 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.814685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 92/175] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:07.820407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19acff-26aa-4ead-ab84-6ad6b9718277 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.821212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19acff-26aa-4ead-ab84-6ad6b9718277 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.821689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb19acff-26aa-4ead-ab84-6ad6b9718277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.842349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fb19acff-26aa-4ead-ab84-6ad6b9718277 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:07.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fb19acff-26aa-4ead-ab84-6ad6b9718277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:07.849054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:07.878932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57959925-b9bc-438d-8daf-6fab8683708d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:07.925812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:07.927139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.927848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.928961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:07.929674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:07.931308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dadf0b8-a408-42ee-bfb6-670082750c42 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:07.932421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 93/176] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:07.933880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6948ca3f-454d-417d-9d3e-fd96a0dbf7ad None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.934246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6948ca3f-454d-417d-9d3e-fd96a0dbf7ad None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.934522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6948ca3f-454d-417d-9d3e-fd96a0dbf7ad None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.936386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 94/177] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:07.937446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6948ca3f-454d-417d-9d3e-fd96a0dbf7ad None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:07.947843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 84/178] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:07.949320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc9e8a1-80a1-4387-9161-57c8eef9fa40 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:07.949615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc9e8a1-80a1-4387-9161-57c8eef9fa40 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:07.949897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc9e8a1-80a1-4387-9161-57c8eef9fa40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:07.971969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1cc9e8a1-80a1-4387-9161-57c8eef9fa40 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:07.973260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1cc9e8a1-80a1-4387-9161-57c8eef9fa40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:08.044017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.054549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 85/179] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:08.055618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.055901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.056173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.074469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6948ca3f-454d-417d-9d3e-fd96a0dbf7ad None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.076603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:08.078096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:08.118166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.118950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.119290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.119474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.121093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:08.123589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:08.130556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 95/180] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:07 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:08.130556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.161473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 86/181] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:08.172447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.172447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.273873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f589dd8d-4591-4712-874f-9c4064877ad3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.308199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f526340b-90ff-4cb1-81d6-db1d9aa5879d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.331639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.332303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.332529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.332800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] PATH_INFO: `/v3/users/fb5de2b0ea574c64b443f39117fbe50c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.333965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] RBAC: Authorizing `identity:get_user(user_id=fb5de2b0ea574c64b443f39117fbe50c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:08.335515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:08.337065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 96/182] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/users/fb5de2b0ea574c64b443f39117fbe50c => generated 275 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:08.337838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.354948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.355603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.355886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.356160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.359716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.368163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:08.375044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:08.376509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:08.377982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 87/183] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:08.381966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.465454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.467036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710c454-24c5-4e13-861d-dce5f357419f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.509790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82c7f8-4e1d-4639-8102-37a739684cf4 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.510271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.510916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.511198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.511476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.512678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:08.514176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:08.520560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 97/184] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:08.520560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.554118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.554728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.555006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.555273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.556013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=fb5de2b0ea574c64b443f39117fbe50c) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:08.556351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:08.557895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:08.565150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 88/185] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/role_assignments?user.id=fb5de2b0ea574c64b443f39117fbe50c&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:08.565150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.625374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.653546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8afe00-2381-4085-b0fd-f4ba9e015e69 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.706372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.707145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.708328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.730695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:08.739601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:08.740628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:08.741398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 98/186] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:08.742184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.745571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110b16f1-4264-4a70-b73f-aa78b58b8058 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.781000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.794999 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.795924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.796885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.797407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.875697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127ad8e0-631c-48d5-9d29-41cb4e699ec4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:08.885959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:08.887446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:08.911385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:08.912006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:08.912319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:08.912638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:08.913942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:08.914500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:08.917344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:08.926631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 99/187] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:08.927820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:08.940115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:08.995007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 89/188] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:08.998933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.027270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.036705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5751ffa-2a1a-411d-8dda-a02ae91de689 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.072458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c4cd0b-2eea-42b8-b714-c7a83890ed49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.073073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c4cd0b-2eea-42b8-b714-c7a83890ed49 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.073355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c4cd0b-2eea-42b8-b714-c7a83890ed49 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.073625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c4cd0b-2eea-42b8-b714-c7a83890ed49 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.074268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c4cd0b-2eea-42b8-b714-c7a83890ed49 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.075874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c4cd0b-2eea-42b8-b714-c7a83890ed49 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.097466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 100/189] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:08 2022] POST /identity/v3/users => generated 297 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:09.098668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943a091-7811-4516-a674-be77656797dc None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.098956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943a091-7811-4516-a674-be77656797dc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.099231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943a091-7811-4516-a674-be77656797dc None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.100721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 101/190] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:09.101404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d943a091-7811-4516-a674-be77656797dc None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.121013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe9814c-d7c5-444e-94a9-7dd99be1da67 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.161603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.162175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.162453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.162724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.163417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:09.163730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.165288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.175118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 90/191] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:09.176030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.190573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.242978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d943a091-7811-4516-a674-be77656797dc None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.284308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.290801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4a7468-3475-45f4-8be6-52f4cd972350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.292290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a7468-3475-45f4-8be6-52f4cd972350 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.292290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a7468-3475-45f4-8be6-52f4cd972350 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.292539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4a7468-3475-45f4-8be6-52f4cd972350 None admin] PATH_INFO: `/v3/users/5457228f44ec4107b64320b303bc16b1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.301156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9da28c6-8e41-4b74-b1e8-5697b09eac0a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.313865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a7468-3475-45f4-8be6-52f4cd972350 None admin] RBAC: Authorizing `identity:get_user(user_id=5457228f44ec4107b64320b303bc16b1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.315836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4a7468-3475-45f4-8be6-52f4cd972350 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.323951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 102/192] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/users/5457228f44ec4107b64320b303bc16b1 => generated 283 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:09.323951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.323951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.323951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.344937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:09.346168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:09.350869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.352028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.352386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.352670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.354606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.356728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.360047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 91/193] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:09.361298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.443489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 103/194] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:09.444347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.495237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5a10c3-a667-4ada-80f7-ff17628d991b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.517595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.535232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.535972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.536393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.536992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] PATH_INFO: `/v3/users/5457228f44ec4107b64320b303bc16b1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.538481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] RBAC: Authorizing `identity:get_user(user_id=5457228f44ec4107b64320b303bc16b1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.539972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.543211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 92/195] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/users/5457228f44ec4107b64320b303bc16b1 => generated 283 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:09.543867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.578646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.586946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f95237-601d-4d60-b611-ffe48b36702b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.626921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.628061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.628288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.628564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.686898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b465361f-71d8-4bb7-b2cb-207b46beee58 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:09.715468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.717986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.730219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:09.730789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:09.731163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:09.731508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:09.732362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] RBAC: Adding query filter params (role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=fb5de2b0ea574c64b443f39117fbe50c) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:09.732744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:09.734658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:09.748276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 93/196] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/role_assignments?user.id=fb5de2b0ea574c64b443f39117fbe50c&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:09.748276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.829850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.929667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 104/197] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:09.930646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:09.944027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:09.962077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ec929-e8dd-410e-94ba-3a45892a5cb2 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.005778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.006443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.006668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.006901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.025946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:10.031364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:10.032334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:10.033181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 94/198] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:10.033859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.091932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782359c8-bb78-400e-b996-fe2d9a6c5250 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.139637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.140270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.140562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.140836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.171311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:10.176926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb13079-2ccc-47b1-b18e-410542863e62 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.226803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.227706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.227939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.228218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.230126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.231992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.234857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 95/199] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:10.235763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.249341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.252386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.258923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:10.361357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad7108b-a0eb-41d7-9bc6-cad735862c01 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.396493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 105/200] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:10.397816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.406873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.407523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.407825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.408098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.408809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:10.409111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.410556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.421406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 96/201] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:10.422367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.476309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:10.527185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:10.531620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1175462-f363-4a44-9989-aaafc55fc99e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.552255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21b567-6f26-4999-aa4c-114e445dc561 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.575045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.575930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.576389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.576926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.577895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=5457228f44ec4107b64320b303bc16b1) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:10.578393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.580254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.587580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 106/202] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/role_assignments?user.id=5457228f44ec4107b64320b303bc16b1&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:10.589235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.592919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.593590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.593869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.595016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.595966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.597713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.599659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 97/203] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:10.605923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.715433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49149754-1a10-4174-851e-c8582e48c05b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.733334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:10.761411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.762411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.762838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.763255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] PATH_INFO: `/v3/users/ac6f6c58b5ff4b85ab98f60d56e451b1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.764629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] RBAC: Authorizing `identity:get_user(user_id=ac6f6c58b5ff4b85ab98f60d56e451b1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.766492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.768340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 107/204] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/users/ac6f6c58b5ff4b85ab98f60d56e451b1 => generated 289 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:10.775073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:10.809212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a412f4c0-8699-4ce5-80d0-82b388ccb0ac None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:10.847652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:10.848593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:10.849024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:10.849438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:10.850322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] RBAC: Adding query filter params (role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=5457228f44ec4107b64320b303bc16b1) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:10.850779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:10.852561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:10.866339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 98/205] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/role_assignments?user.id=5457228f44ec4107b64320b303bc16b1&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 259 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:10.969381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477bc1b7-db10-4479-9eb0-c8f6d4c50274 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:11.031189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:11.032308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.032821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.033317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.102244 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:11.123574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:11.125203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:11.245233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 108/206] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:11.246746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:11.279544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:11.435466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618cb2d-cf65-4464-860a-f00f0ae94a1c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:11.482091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cb4474-a109-490b-be5c-84ff9d8d325a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:11.482752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cb4474-a109-490b-be5c-84ff9d8d325a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.483033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cb4474-a109-490b-be5c-84ff9d8d325a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.483308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cb4474-a109-490b-be5c-84ff9d8d325a None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.484545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cb4474-a109-490b-be5c-84ff9d8d325a None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:11.486004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cb4474-a109-490b-be5c-84ff9d8d325a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:11.487979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 99/207] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:11 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:11.494025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:11.593894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:11.615740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f819bf50-6774-4b19-ae9d-8517ee0a8b89 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:11.634058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7380fc-0f81-4208-977f-bf2a45b1d7cb None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.634396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7380fc-0f81-4208-977f-bf2a45b1d7cb None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.634624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7380fc-0f81-4208-977f-bf2a45b1d7cb None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.636399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 100/208] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:55:11.642526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbe2c0-2ae1-46fa-bf42-7c8d9d881526 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.643167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbe2c0-2ae1-46fa-bf42-7c8d9d881526 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.643734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcbe2c0-2ae1-46fa-bf42-7c8d9d881526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.659037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:11.660268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.660268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.660268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.661021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] RBAC: Adding query filter params (role.id=00bbfd499c354c379adeca6c2d8ae65e, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3, user.id=ac6f6c58b5ff4b85ab98f60d56e451b1) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:11.661700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:11.663646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659e8a4-3a33-4bf5-a9e6-ab750bb00681 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:11.672294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 109/209] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:11 2022] GET /identity/v3/role_assignments?user.id=ac6f6c58b5ff4b85ab98f60d56e451b1&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=00bbfd499c354c379adeca6c2d8ae65e => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:11.680823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-abcbe2c0-2ae1-46fa-bf42-7c8d9d881526 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:11.682850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-abcbe2c0-2ae1-46fa-bf42-7c8d9d881526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:11.802287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 101/210] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:11.832219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94acff-cda7-4dab-8e0b-437633d77344 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.832558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94acff-cda7-4dab-8e0b-437633d77344 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.832812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94acff-cda7-4dab-8e0b-437633d77344 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.834545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 110/211] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:11.841077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:11.841400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:11.841668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:11.850465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e94acff-cda7-4dab-8e0b-437633d77344 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:11.870014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:11.871243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:11.873706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:11.959376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 102/212] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:11.995012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:12.023836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e94acff-cda7-4dab-8e0b-437633d77344 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:12.108180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:12.108868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:12.109144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:12.109419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:12.113395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:12.114946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:12.117956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 111/213] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:12.124915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:12.171179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:12.248732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79b10c6-ff11-49a3-8d33-eaaddef57fdf None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:12.341548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:12.342207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:12.342496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:12.342774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:12.359576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:12.363090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d69635-0b98-4d3f-af13-3b8d51c45db7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:12.368695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:12.384794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:12.385882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:12.386659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 103/214] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 390 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:12.397320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:12.457209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:12.457933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:12.458176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:12.458446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:12.461038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:12.462187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:12.500666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 112/215] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:12 2022] POST /identity/v3/users => generated 297 bytes in 376 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:55:12.585203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f153d3d-35c4-46f5-ace0-910615dd0a61 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:12.633391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:12.634417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:12.634905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:12.635448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:12.636469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:12.636988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:12.638763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:12.664871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 104/216] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:12.668834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:12.770253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:12.820457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb19539-b877-4910-8253-c31207fccb1c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:12.866612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c9fa70-bfa0-4118-b3ef-f8ed3225067a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:12.867184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c9fa70-bfa0-4118-b3ef-f8ed3225067a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:12.867468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c9fa70-bfa0-4118-b3ef-f8ed3225067a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:12.867738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c9fa70-bfa0-4118-b3ef-f8ed3225067a None admin] PATH_INFO: `/v3/users/7fb0039654844864bcca93bf6840679f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:12.890798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c9fa70-bfa0-4118-b3ef-f8ed3225067a None admin] RBAC: Authorizing `identity:get_user(user_id=7fb0039654844864bcca93bf6840679f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:12.892386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c9fa70-bfa0-4118-b3ef-f8ed3225067a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:12.894439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 113/217] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:12 2022] GET /identity/v3/users/7fb0039654844864bcca93bf6840679f => generated 297 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:12.901523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:13.022865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:13.097547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3ee8e6-e28e-41b8-99b3-d06e2cca97b8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:13.142022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:13.142536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.142821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.143109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.183268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca9e0a-9fc9-4776-92d9-424974c670b9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.183583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca9e0a-9fc9-4776-92d9-424974c670b9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.183825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca9e0a-9fc9-4776-92d9-424974c670b9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.186103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 114/218] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:13.191351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.191639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.191919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.218398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:13.219615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:13.245755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:13.247810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:13.320264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 115/219] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:13.336402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:13.341669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:13.401236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:13.425250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 105/220] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:13.431046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:13.449557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92e8934-eea8-4e05-af59-142b662d64e1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:13.489057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e239c74-aae2-4ffa-9bad-d20ad1c7e300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:13.490098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e239c74-aae2-4ffa-9bad-d20ad1c7e300 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.490444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e239c74-aae2-4ffa-9bad-d20ad1c7e300 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.490795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e239c74-aae2-4ffa-9bad-d20ad1c7e300 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.508740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e239c74-aae2-4ffa-9bad-d20ad1c7e300 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-7e239c74-aae2-4ffa-9bad-d20ad1c7e300 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:13.513864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:13.514806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:13.515596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 116/221] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:13.520347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a129895-8e52-4e62-82d0-d96668d056c8 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.521794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a129895-8e52-4e62-82d0-d96668d056c8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a129895-8e52-4e62-82d0-d96668d056c8 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.524342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 117/222] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:13 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:13.528354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a129895-8e52-4e62-82d0-d96668d056c8 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:13.571556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b4e87-286a-4928-8150-b718805f7990 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:13.611729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ff9f8a-2459-4897-b57a-b7d910e29fdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:13.612793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff9f8a-2459-4897-b57a-b7d910e29fdb None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.613284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff9f8a-2459-4897-b57a-b7d910e29fdb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.613810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff9f8a-2459-4897-b57a-b7d910e29fdb None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.615265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ff9f8a-2459-4897-b57a-b7d910e29fdb None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:13.617216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ff9f8a-2459-4897-b57a-b7d910e29fdb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:13.620064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 106/223] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:13 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:13.621635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.622150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.622623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.642829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:13.644522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:13.660846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a129895-8e52-4e62-82d0-d96668d056c8 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:13.676651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:13.699023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:13.730176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:13.731156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.731689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.732160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.733518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:13.734048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:13.736628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:13.748412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 118/224] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:13.752311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:13.772997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 107/225] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:13.782018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:13.878741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7960c-d4aa-47b5-8682-557095dc155e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:13.931607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:13.932885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:13.933167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:13.933443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:13.935142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=7fb0039654844864bcca93bf6840679f) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:13.936004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:13.939591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:13.948315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 119/226] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:13 2022] GET /identity/v3/role_assignments?user.id=7fb0039654844864bcca93bf6840679f&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:13.954730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:14.018055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c772a-5eb6-4f0c-87c4-fe112fb1cfb5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:14.088657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:14.161436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3842b32b-928c-4983-97b0-af8916603f82 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:14.171697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:14.172674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:14.173089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:14.173490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] PATH_INFO: `/v3/users/5457228f44ec4107b64320b303bc16b1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:14.183727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] RBAC: Authorizing `identity:get_user(user_id=5457228f44ec4107b64320b303bc16b1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:14.185965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:14.188803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 108/227] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:13 2022] GET /identity/v3/users/5457228f44ec4107b64320b303bc16b1 => generated 283 bytes in 415 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:14.198235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:14.211707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:14.212696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:14.214590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:14.215047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:14.231909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:14.239051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:14.240406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:14.241854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 120/228] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:14.249902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:14.260941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:14.356954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e0e5d7-bb5d-4c5d-b9ad-b526250260b9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:14.425201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:14.426504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:14.426782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:14.429415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:14.430774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:14.432421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:14.440203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 109/229] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:14 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:14.441049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:14.544835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a626f0e-ab36-406e-893b-e4de5ee2b5a5 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:14.556625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:14.600754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:14.616831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5649d28-b65d-445e-8a6f-5825019cc575 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:14.676036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:14.677269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:14.677686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:14.677969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/5457228f44ec4107b64320b303bc16b1/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:14.680623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] RBAC: Authorizing `identity:create_grant(project_id=95caef1c516c49b598639b368c2ec7ad, user_id=5457228f44ec4107b64320b303bc16b1, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:14.682899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:14.689045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:14.690114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:14.690549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:14.690802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:14.691692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:14.692067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:14.693705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:14.703368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 110/230] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:14 2022] PUT /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/5457228f44ec4107b64320b303bc16b1/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:14.723921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 121/231] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 474 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:14.731489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:14.870628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d961e783-ef05-444d-aa6e-0b36cb8ddefb None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:14.920818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:14.921929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:14.922215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:14.922517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] PATH_INFO: `/v3/users/5457228f44ec4107b64320b303bc16b1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:14.924156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] RBAC: Authorizing `identity:get_user(user_id=5457228f44ec4107b64320b303bc16b1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:14.925716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:14.927799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 111/232] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:14 2022] GET /identity/v3/users/5457228f44ec4107b64320b303bc16b1 => generated 283 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:14.937483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:15.168614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2758580d-2318-48b2-98e0-418c6ae2f077 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:15.244733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5b50be-09dc-43b5-baf7-f27f0cffb1c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:15.245849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b50be-09dc-43b5-baf7-f27f0cffb1c4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:15.246394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b50be-09dc-43b5-baf7-f27f0cffb1c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:15.246872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5b50be-09dc-43b5-baf7-f27f0cffb1c4 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:15.248922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5b50be-09dc-43b5-baf7-f27f0cffb1c4 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:15.251645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5b50be-09dc-43b5-baf7-f27f0cffb1c4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:15.255086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 122/233] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:14 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:15.264837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:15.316635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:15.446299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c884cad-8a0c-4ca6-ab09-e5822ef0d736 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:15.499161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1822a8ce-2cbf-48d3-ad12-59b21fad47b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:15.500230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822a8ce-2cbf-48d3-ad12-59b21fad47b4 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:15.500774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822a8ce-2cbf-48d3-ad12-59b21fad47b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:15.501230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822a8ce-2cbf-48d3-ad12-59b21fad47b4 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/5457228f44ec4107b64320b303bc16b1/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:15.506293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1822a8ce-2cbf-48d3-ad12-59b21fad47b4 None admin] RBAC: Authorizing `identity:create_grant(project_id=95caef1c516c49b598639b368c2ec7ad, user_id=5457228f44ec4107b64320b303bc16b1, role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:15.508211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1822a8ce-2cbf-48d3-ad12-59b21fad47b4 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:15.531726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 112/234] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:15 2022] PUT /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/5457228f44ec4107b64320b303bc16b1/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 00:55:15.732154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0730da-6bf2-40a9-a0e5-5f42f6e03e36 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:15.732839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0730da-6bf2-40a9-a0e5-5f42f6e03e36 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:15.732839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0730da-6bf2-40a9-a0e5-5f42f6e03e36 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:15.734527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 123/235] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:15.740355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb287a-70d7-43c4-951d-2dcb92afe739 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:15.740664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb287a-70d7-43c4-951d-2dcb92afe739 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:15.740914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb287a-70d7-43c4-951d-2dcb92afe739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:15.764887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-0efb287a-70d7-43c4-951d-2dcb92afe739 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:15.776121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0efb287a-70d7-43c4-951d-2dcb92afe739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:15.901566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 113/236] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:15.932520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0730da-6bf2-40a9-a0e5-5f42f6e03e36 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:15.953143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bfc757-ad33-41a8-8de6-b675e117f29e None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:15.953497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bfc757-ad33-41a8-8de6-b675e117f29e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:15.953825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bfc757-ad33-41a8-8de6-b675e117f29e None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:15.955555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 114/237] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:15.962306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:15.962609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:15.962902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:15.990770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:15.994901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:15.996603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:16.011861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:16.183077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 115/238] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:16.209858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0730da-6bf2-40a9-a0e5-5f42f6e03e36 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:16.210984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:16.305299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:16.307328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:16.310369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:16.310860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:16.355811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:16.369044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:16.371652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:16.375072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:16.379633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 124/239] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 448 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:16.393388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:16.442085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad0996d-ad50-4d89-bda5-7d4f4ab3d388 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:16.558740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:16.559749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:16.560270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:16.560792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:16.565058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:16.567906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:16.574873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 116/240] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:16.582518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:16.623529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:16.733048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:16.757550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d19da13-eefc-4de6-bba9-939481b767b0 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:16.841824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518419fe-3614-447f-9454-00a9004708ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:16.843754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518419fe-3614-447f-9454-00a9004708ce None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:16.844259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518419fe-3614-447f-9454-00a9004708ce None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:16.844807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518419fe-3614-447f-9454-00a9004708ce None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:16.846139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518419fe-3614-447f-9454-00a9004708ce None admin] RBAC: Adding query filter params (name=reader) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:16.846705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518419fe-3614-447f-9454-00a9004708ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:16.848812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518419fe-3614-447f-9454-00a9004708ce None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:16.859520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 125/241] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 467 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:16.871517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518419fe-3614-447f-9454-00a9004708ce None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:16.941590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb642258-1311-4055-9473-e95a68d4eb6f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:17.056056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518419fe-3614-447f-9454-00a9004708ce None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:17.105643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:17.106879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.107394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.107865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] PATH_INFO: `/v3/users/b582651f7d3042a9a5e612ebfdb178fc` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.121993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8429ca7-2c70-4296-be2f-8d8b9a53cbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:17.123206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8429ca7-2c70-4296-be2f-8d8b9a53cbbe None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.123776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8429ca7-2c70-4296-be2f-8d8b9a53cbbe None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.124361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8429ca7-2c70-4296-be2f-8d8b9a53cbbe None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.126055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8429ca7-2c70-4296-be2f-8d8b9a53cbbe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:17.129000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] RBAC: Authorizing `identity:get_user(user_id=b582651f7d3042a9a5e612ebfdb178fc)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:17.130869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:17.138527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 126/242] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:16 2022] GET /identity/v3/users/b582651f7d3042a9a5e612ebfdb178fc => generated 297 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:17.139957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8429ca7-2c70-4296-be2f-8d8b9a53cbbe None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:17.140572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:17.160471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:17.228528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 117/243] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:16 2022] POST /identity/v3/users => generated 293 bytes in 641 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:55:17.331034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d60b0-562f-4316-af43-be54be8265ae None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.331591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d60b0-562f-4316-af43-be54be8265ae None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.332154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d60b0-562f-4316-af43-be54be8265ae None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.341359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed91267-5502-443a-9366-17558cdf04a3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:17.347125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 118/244] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:17 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:17.374405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef37e149-d99a-4d22-ba10-5ce012f043e3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.374817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef37e149-d99a-4d22-ba10-5ce012f043e3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.375311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef37e149-d99a-4d22-ba10-5ce012f043e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.448048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ef37e149-d99a-4d22-ba10-5ce012f043e3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:17.451013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ef37e149-d99a-4d22-ba10-5ce012f043e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:17.499188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce97507-4b45-4c76-b0b8-99169b73ab0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:17.500106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce97507-4b45-4c76-b0b8-99169b73ab0f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.500728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce97507-4b45-4c76-b0b8-99169b73ab0f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.501512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce97507-4b45-4c76-b0b8-99169b73ab0f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.608406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce97507-4b45-4c76-b0b8-99169b73ab0f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:17.610518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce97507-4b45-4c76-b0b8-99169b73ab0f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:17.625699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:17.634094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:17.649645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 119/245] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:17.691567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0c16c3-6d13-4cb0-a62e-d3df1263f2c2 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.692101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0c16c3-6d13-4cb0-a62e-d3df1263f2c2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.693255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0c16c3-6d13-4cb0-a62e-d3df1263f2c2 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.696840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 120/246] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:55:17.705800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0c16c3-6d13-4cb0-a62e-d3df1263f2c2 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:17.806639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 127/247] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 667 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 07 00:55:17.809536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.810325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.810325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:17.839449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:17.841473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:17.933087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0c16c3-6d13-4cb0-a62e-d3df1263f2c2 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:17.962287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 128/248] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:17.963295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:17.985968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:17.987494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:17.988015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:17.989014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:18.014865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:18.021974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:18.022968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:18.023858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:18.027175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 121/249] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:18.028729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:18.079659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:18.141798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:18.236894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa45096-9d8d-4cd7-be0d-ddc830d72c26 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:18.278327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61042430-6d7a-4322-b499-20c7c9a0fe56 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:18.283180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:18.292987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:18.292987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:18.292987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:18.292987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:18.292987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:18.292987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 129/250] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:17 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:18.293556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:18.385397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:18.386116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:18.386886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:18.387307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:18.418825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:18.425453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:18.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:18.427246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 122/251] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 399 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:18.429202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:18.433459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdf129d-f9ee-4072-95ea-81378c48138c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:18.464640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:18.504156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:18.508477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:18.509122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:18.509606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:18.511175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:18.512286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:18.515408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:18.529634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 130/252] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:18.530908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:18.686325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3439cf6c-a05c-46aa-9405-c2cb0009a7d8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:18.706847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdefd29-86e4-4f51-b3a6-714f70624d42 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:18.733536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:18.755271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:18.756451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:18.756968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:18.758030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:18.759336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:18.760676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:18.762640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:18.773256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 131/253] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:18.774683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:18.793701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:18.794376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:18.794649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:18.794925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:18.795674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] RBAC: Adding query filter params (role.id=00bbfd499c354c379adeca6c2d8ae65e, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=b582651f7d3042a9a5e612ebfdb178fc) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:18.796045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:18.797629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:18.813429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 123/254] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:18 2022] GET /identity/v3/role_assignments?user.id=b582651f7d3042a9a5e612ebfdb178fc&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=00bbfd499c354c379adeca6c2d8ae65e => generated 259 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:18.814388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:18.836716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:19.028360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6115e8b4-f921-4325-b5f0-3d73bd475e7b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.046229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ad27dd-437b-41ac-abf3-def52326656a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.092651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f507ead2-d784-4a57-9f79-bfb61ecf519b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:19.092893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8401656-57b8-4c68-a98f-e8920073fc5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:19.093432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f507ead2-d784-4a57-9f79-bfb61ecf519b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.093539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8401656-57b8-4c68-a98f-e8920073fc5a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.093766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f507ead2-d784-4a57-9f79-bfb61ecf519b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.093864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8401656-57b8-4c68-a98f-e8920073fc5a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.094114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f507ead2-d784-4a57-9f79-bfb61ecf519b None admin] PATH_INFO: `/v3/users/7fb0039654844864bcca93bf6840679f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.094212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8401656-57b8-4c68-a98f-e8920073fc5a None admin] PATH_INFO: `/v3/users/5457228f44ec4107b64320b303bc16b1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.095379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f507ead2-d784-4a57-9f79-bfb61ecf519b None admin] RBAC: Authorizing `identity:get_user(user_id=7fb0039654844864bcca93bf6840679f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:19.097051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f507ead2-d784-4a57-9f79-bfb61ecf519b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:19.099282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 132/255] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:18 2022] GET /identity/v3/users/7fb0039654844864bcca93bf6840679f => generated 297 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:19.100670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb63402-b08e-4e7f-93c9-6b77aa217188 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.100961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb63402-b08e-4e7f-93c9-6b77aa217188 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.101237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb63402-b08e-4e7f-93c9-6b77aa217188 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.102052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8401656-57b8-4c68-a98f-e8920073fc5a None admin] RBAC: Authorizing `identity:get_user(user_id=5457228f44ec4107b64320b303bc16b1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:19.102900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 133/256] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:19.103782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8401656-57b8-4c68-a98f-e8920073fc5a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:19.104565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb63402-b08e-4e7f-93c9-6b77aa217188 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:19.105777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 124/257] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:18 2022] GET /identity/v3/users/5457228f44ec4107b64320b303bc16b1 => generated 283 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:19.111753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.112139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.112408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.134991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:19.136351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:19.255320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb63402-b08e-4e7f-93c9-6b77aa217188 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.258109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:19.271059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:19.302795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:19.304187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.304603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.305032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.309745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:19.310905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:19.313075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 134/258] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:19 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:19.314161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:19.341552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 125/259] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:19.342419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:19.452188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9954e085-4722-4dc0-a6d6-8ef3a04152ae None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.492575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:19.493563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.494042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.494557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.552284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:19.589336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:19.590977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:19.619887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fc6572-a0d8-4f8f-a6a9-b13280a1fc57 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.697039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:19.698558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.698558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.699003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/7fb0039654844864bcca93bf6840679f/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.700900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] RBAC: Authorizing `identity:create_grant(project_id=95caef1c516c49b598639b368c2ec7ad, user_id=7fb0039654844864bcca93bf6840679f, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:19.702367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:19.707088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 135/260] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:19.708588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:19.734893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 126/261] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:19 2022] PUT /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/7fb0039654844864bcca93bf6840679f/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:19.735595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:19.812833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:19.816044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:19.859042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442104b-bedf-4ba1-abed-ad27d7c05432 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.911029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:19.911692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:19.911972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:19.912284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:19.943250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:19.950415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be978be7-38ec-46ad-8e17-fbe59592061e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:19.955302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:19.956514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:19.957483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 136/262] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:19.967104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:20.041040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:20.041786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.042075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.042631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.044539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:20.046740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:20.048905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 127/263] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:19 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:20.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:20.187337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226adaf5-f4cc-480e-823c-76375d224c0a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:20.242396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:20.250636 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:20.251408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.251721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.252005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.253270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:20.253270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:20.254949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:20.267482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 137/264] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:20.274004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:20.280205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a49232-88d1-49a4-aec8-15ada527e5e6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:20.298755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:20.341695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:20.342731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.343202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.343656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.344719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=5457228f44ec4107b64320b303bc16b1) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:20.345158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:20.347212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:20.358994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 128/265] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:20 2022] GET /identity/v3/role_assignments?user.id=5457228f44ec4107b64320b303bc16b1&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:20.471680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de892ef7-ac80-4c06-84e7-c14cc0f70eaa None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:20.517491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa17e87-db5a-4d64-9d77-bb3c8dc5aeba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:20.518166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa17e87-db5a-4d64-9d77-bb3c8dc5aeba None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.518464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa17e87-db5a-4d64-9d77-bb3c8dc5aeba None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.518740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa17e87-db5a-4d64-9d77-bb3c8dc5aeba None admin] PATH_INFO: `/v3/users/5457228f44ec4107b64320b303bc16b1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.520067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa17e87-db5a-4d64-9d77-bb3c8dc5aeba None admin] RBAC: Authorizing `identity:get_user(user_id=5457228f44ec4107b64320b303bc16b1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:20.521641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa17e87-db5a-4d64-9d77-bb3c8dc5aeba None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:20.523911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 138/266] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:20 2022] GET /identity/v3/users/5457228f44ec4107b64320b303bc16b1 => generated 283 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:20.530290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:20.686358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c54270-1dc5-4b70-a1a1-a360be8ec108 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.686534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c54270-1dc5-4b70-a1a1-a360be8ec108 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.686814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c54270-1dc5-4b70-a1a1-a360be8ec108 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.688373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 139/267] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:20.694164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.694494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.694717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.712755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92a44e-e698-4ca2-a832-f59c3789265b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:20.733980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:20.735671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:20.822030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198d2786-eefa-45fe-a6ac-335f5eb6b2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:20.823615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198d2786-eefa-45fe-a6ac-335f5eb6b2b6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:20.824508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198d2786-eefa-45fe-a6ac-335f5eb6b2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:20.825357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198d2786-eefa-45fe-a6ac-335f5eb6b2b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:20.889058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:20.916189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:20.926862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198d2786-eefa-45fe-a6ac-335f5eb6b2b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:20.930166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198d2786-eefa-45fe-a6ac-335f5eb6b2b6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:20.940977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 140/268] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:20.969676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.096590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf26b8a0-0cf8-4f3f-964e-aaa63047d895 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.113864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 129/269] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 584 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:21.116868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9231c75-d12a-475c-9242-f54202b9e94d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.116868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9231c75-d12a-475c-9242-f54202b9e94d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.116868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9231c75-d12a-475c-9242-f54202b9e94d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.120339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 130/270] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:21.124318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9231c75-d12a-475c-9242-f54202b9e94d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.144860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8299df-fb01-41cf-9061-8420f6a7904f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.145215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8299df-fb01-41cf-9061-8420f6a7904f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.148579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8299df-fb01-41cf-9061-8420f6a7904f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.148579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8299df-fb01-41cf-9061-8420f6a7904f None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.163078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8299df-fb01-41cf-9061-8420f6a7904f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-9f8299df-fb01-41cf-9061-8420f6a7904f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:21.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:21.169799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:21.170591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:21.173013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 141/271] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:21.174656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.175151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.175676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.196651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:21.198838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:21.238892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:21.257668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:21.287110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9231c75-d12a-475c-9242-f54202b9e94d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.304645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 142/272] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:21.306738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.328888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.329901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.330375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.330861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.332296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:21.334004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:21.337131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 131/273] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:21 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:21.343261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.461912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8e629b-d80f-4ecd-9475-3567b7e0cc2b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.501032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.501648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.501974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.502246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.503192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:21.503668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:21.505419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:21.515214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 143/274] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:21.516479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.531468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52459f4-d23e-43f8-ad15-47e9b6ad6e59 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.575126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:21.590655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.592185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.592639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.593351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.594261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] RBAC: Adding query filter params (role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=5457228f44ec4107b64320b303bc16b1) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:21.594962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:21.597345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:21.607229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 132/275] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:21 2022] GET /identity/v3/role_assignments?user.id=5457228f44ec4107b64320b303bc16b1&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 608 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:21.608475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.612528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:21.655926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da318ff-0c41-43f4-a55d-9d3161a0b818 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.707055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.708559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.709000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.709414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.711685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:21.713764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:21.716405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 144/276] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:21.724586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.761058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c2c42-1249-4ed8-8bb0-7ace7a27f3b8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.837255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.837977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.838339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.838618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] PATH_INFO: `/v3/users/8a19427696d94a4fa4987bb65cfd9e8e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.861140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] RBAC: Authorizing `identity:get_user(user_id=8a19427696d94a4fa4987bb65cfd9e8e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:21.862384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:21.865205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 133/277] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:21 2022] GET /identity/v3/users/8a19427696d94a4fa4987bb65cfd9e8e => generated 293 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:21.871148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:21.910098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed67167-0c89-4048-aaea-4a29b453fd95 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:21.934434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:21.970831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:21.974562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0a5cb4-9b7d-4259-9024-f8952c47dff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:21.975876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0a5cb4-9b7d-4259-9024-f8952c47dff4 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:21.976615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0a5cb4-9b7d-4259-9024-f8952c47dff4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:21.977347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0a5cb4-9b7d-4259-9024-f8952c47dff4 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:21.981097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0a5cb4-9b7d-4259-9024-f8952c47dff4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:21.982621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0a5cb4-9b7d-4259-9024-f8952c47dff4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:22.017925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 145/278] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:21 2022] POST /identity/v3/users => generated 293 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:55:22.023652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c9c129-ab07-486b-a853-9f4db9c03894 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:22.068110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:22.069161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:22.069648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:22.070126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:22.071246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, user.id=8a19427696d94a4fa4987bb65cfd9e8e, scope.system=all) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:22.072075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:22.073986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:22.078279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67b9910-b959-4521-a5d6-023949f57ccf None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:22.078612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67b9910-b959-4521-a5d6-023949f57ccf None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:22.078897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67b9910-b959-4521-a5d6-023949f57ccf None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:22.081152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 146/279] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:55:22.081850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 134/280] 10.222.0.26 () {56 vars in 1308 bytes} [Sun Aug 7 00:55:21 2022] GET /identity/v3/role_assignments?user.id=8a19427696d94a4fa4987bb65cfd9e8e&scope.system=all&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:22.087418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:22.087726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:22.088519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:22.114633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:22.116387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:22.204177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 147/281] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:22.228174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:22.364970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c2efb6-02b7-4f27-b237-c83996c52ae9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:22.396256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:22.410020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:22.410671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:22.410994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:22.411270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:22.423222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:22.428275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:22.429467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:22.430277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 135/282] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:22.436273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:22.553161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01677be0-da5b-4992-9286-bf63a0755e56 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:22.558165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:22.619593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:22.620348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:22.620681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:22.620979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:22.621773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:22.622086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:22.623690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:22.640416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 148/283] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:22.646211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:22.831418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917f0082-785b-4f66-bbd9-846d214fb3fe None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:22.873200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a693290-2126-478d-a349-30753778ee3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:22.873834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a693290-2126-478d-a349-30753778ee3b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:22.874122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a693290-2126-478d-a349-30753778ee3b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:22.874402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a693290-2126-478d-a349-30753778ee3b None admin] PATH_INFO: `/v3/users/b582651f7d3042a9a5e612ebfdb178fc` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:22.881040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a693290-2126-478d-a349-30753778ee3b None admin] RBAC: Authorizing `identity:get_user(user_id=b582651f7d3042a9a5e612ebfdb178fc)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:22.882550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a693290-2126-478d-a349-30753778ee3b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:22.884602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 136/284] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:22 2022] GET /identity/v3/users/b582651f7d3042a9a5e612ebfdb178fc => generated 297 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:22.890120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:23.133335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c38f2dd-3ca2-4791-b021-2ec583a4fb25 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:23.153210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0b206-e89c-4e54-9124-9fe6b6f11ab1 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:23.153564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0b206-e89c-4e54-9124-9fe6b6f11ab1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:23.153760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb0b206-e89c-4e54-9124-9fe6b6f11ab1 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:23.155402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 137/285] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:23.168637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:23.168930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:23.169198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:23.192986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:23.194013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:23.194486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:23.194946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:23.196986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:23.198774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:23.199177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:23.200600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:23.200796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 149/286] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:22 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:23.206228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:23.255173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:23.297342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:23.345356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 138/287] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:23.381443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6831cc1f-77b0-4580-bd44-16e071b33cd9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:23.402467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:23.448320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:23.450345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:23.451999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:23.452645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/b582651f7d3042a9a5e612ebfdb178fc/roles/00bbfd499c354c379adeca6c2d8ae65e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:23.456488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] RBAC: Authorizing `identity:create_grant(project_id=95caef1c516c49b598639b368c2ec7ad, user_id=b582651f7d3042a9a5e612ebfdb178fc, role_id=00bbfd499c354c379adeca6c2d8ae65e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:23.458422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:23.489434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 150/288] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:55:23 2022] PUT /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad/users/b582651f7d3042a9a5e612ebfdb178fc/roles/00bbfd499c354c379adeca6c2d8ae65e => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 00:55:23.564936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3919b1f-9f00-423a-9cf2-14b38be15064 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:23.625583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:23.627029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:23.627728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:23.628219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:23.652182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:23.660078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:23.661508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:23.662631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:23.666105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 139/289] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:23.673381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:23.795865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:23.822518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b845e8db-71ed-43ff-ac07-706f16920d51 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:23.886766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:23.888347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:23.889117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:23.889873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:23.891376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:23.892213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:23.895563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:23.913413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 151/290] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:23.922083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:24.000397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:24.083785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d12329-1e5f-484b-bd22-523be48a943d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:24.129719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294bcc28-cb3f-4dd7-a0a1-06d072697ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:24.130724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294bcc28-cb3f-4dd7-a0a1-06d072697ec9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:24.131197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294bcc28-cb3f-4dd7-a0a1-06d072697ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:24.131658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294bcc28-cb3f-4dd7-a0a1-06d072697ec9 None admin] PATH_INFO: `/v3/users/7fb0039654844864bcca93bf6840679f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:24.133086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294bcc28-cb3f-4dd7-a0a1-06d072697ec9 None admin] RBAC: Authorizing `identity:get_user(user_id=7fb0039654844864bcca93bf6840679f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:24.135354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294bcc28-cb3f-4dd7-a0a1-06d072697ec9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:24.137256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 140/291] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:23 2022] GET /identity/v3/users/7fb0039654844864bcca93bf6840679f => generated 297 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:24.145911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:24.292999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342465e3-9bbb-47e5-85c0-7e0b242af051 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:24.346471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:24.347458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:24.347935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:24.348422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:24.372775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:24.444672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:24.446820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:24.632000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c832288a-8282-40ed-8a3c-52ad7882fae0 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:24.632389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c832288a-8282-40ed-8a3c-52ad7882fae0 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:24.632864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c832288a-8282-40ed-8a3c-52ad7882fae0 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:24.634139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 152/292] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:24.635386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 141/293] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:24.640565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:24.643265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:24.643700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:24.644329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:24.670308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:24.672351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:24.717223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:24.822802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 142/294] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:24.847418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ab2cef-2a95-469e-92f8-533235ce0fcc None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:24.877114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:24.915594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:24.916723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:24.917239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:24.918019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:24.920691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:24.924279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:24.928702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 153/295] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:24 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:24.937952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:24.985463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:25.080029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1ac8d9-7a50-4f21-88ce-d5c09885c9e1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.112008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c0bcd0-168f-408d-b70a-7d699755e7a7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.121971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee76b084-3f63-46ee-8db9-44365003a796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:25.122818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76b084-3f63-46ee-8db9-44365003a796 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.123107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76b084-3f63-46ee-8db9-44365003a796 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.123434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76b084-3f63-46ee-8db9-44365003a796 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.136036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:25.136348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee76b084-3f63-46ee-8db9-44365003a796 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-ee76b084-3f63-46ee-8db9-44365003a796 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:25.142417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:25.143572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:25.144486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:25.146863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 143/296] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:25.148519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d32a3c-b5e6-4926-ae5b-951ebfd0b5e8 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.149025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d32a3c-b5e6-4926-ae5b-951ebfd0b5e8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.149513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d32a3c-b5e6-4926-ae5b-951ebfd0b5e8 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.151674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 144/297] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:25.157014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d32a3c-b5e6-4926-ae5b-951ebfd0b5e8 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:25.164422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:25.165088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.165527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.165958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.167607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=7fb0039654844864bcca93bf6840679f) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:25.168155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:25.170765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dddb9cde-de40-49ab-91de-cba79b1262fe None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:25.182081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 154/298] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:24 2022] GET /identity/v3/role_assignments?user.id=7fb0039654844864bcca93bf6840679f&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 608 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:25.183086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c5d66-7b7e-402f-a177-a24fd128884b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.188274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c5d66-7b7e-402f-a177-a24fd128884b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.188274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c5d66-7b7e-402f-a177-a24fd128884b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.211516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a52c5d66-7b7e-402f-a177-a24fd128884b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:25.214153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a52c5d66-7b7e-402f-a177-a24fd128884b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:25.330637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 155/299] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:25.340536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c36e8-4962-42f0-b675-ce7d4bee9ab2 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.340800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c36e8-4962-42f0-b675-ce7d4bee9ab2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.341275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812c36e8-4962-42f0-b675-ce7d4bee9ab2 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.345334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 156/300] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:25 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:25.355798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.356684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.357048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.379035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:25.380418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:25.405487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:25.415167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d32a3c-b5e6-4926-ae5b-951ebfd0b5e8 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.468888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:25.469252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.469553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.469894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.470675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:25.471076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:25.472677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:25.483968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 145/301] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:25.489158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:25.500565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 157/302] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:25.500752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:25.563043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:25.651452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a42149-745b-4392-9d9d-5e5eeb67a1f0 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.676600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a5188d-ae31-425f-b335-6153058e9ab7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.699617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:25.700284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.700566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.700865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.722852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:25.729941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:25.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 146/303] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:25.733650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:25.758636 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:25.759357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.759671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.760554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] PATH_INFO: `/v3/users/8a19427696d94a4fa4987bb65cfd9e8e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.771439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] RBAC: Authorizing `identity:get_user(user_id=8a19427696d94a4fa4987bb65cfd9e8e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:25.773634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:25.775742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 158/304] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:25 2022] GET /identity/v3/users/8a19427696d94a4fa4987bb65cfd9e8e => generated 293 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:25.776410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:25.790285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:25.932815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a72d70-0e81-4160-9738-cd12621be7d2 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.936861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39bf1d1-027a-4dfe-b5c7-16ac597ebc53 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:25.950471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:25.986117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:25.987188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:25.987678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:25.988165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:25.990471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:25.993390 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:26.002516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 147/305] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:26.002709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:26.002709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.002709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.002953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:26.003031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.003031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] RBAC: Adding query filter params (name=reader) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:26.003031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:26.003459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:26.020051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 159/306] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:26.024482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:26.154895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:26.171446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95c0fa8-a813-44bd-9e2b-2a6495f722d9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:26.204766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f857ea-741b-4a35-a571-5a32475b6ffa None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:26.221932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:26.222643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.222949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.223256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] PATH_INFO: `/v3/system/users/8a19427696d94a4fa4987bb65cfd9e8e/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.224975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=8a19427696d94a4fa4987bb65cfd9e8e, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:26.226677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:26.239961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 148/307] 10.222.0.26 () {58 vars in 1334 bytes} [Sun Aug 7 00:55:25 2022] PUT /identity/v3/system/users/8a19427696d94a4fa4987bb65cfd9e8e/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:26.241272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:26.250465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fb0469-751c-4be8-97d8-36f9040784c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:26.251460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb0469-751c-4be8-97d8-36f9040784c0 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.251854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb0469-751c-4be8-97d8-36f9040784c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.252189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb0469-751c-4be8-97d8-36f9040784c0 None admin] PATH_INFO: `/v3/groups` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.253307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fb0469-751c-4be8-97d8-36f9040784c0 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:26.255452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fb0469-751c-4be8-97d8-36f9040784c0 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:26.296351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 160/308] 10.222.0.26 () {60 vars in 1134 bytes} [Sun Aug 7 00:55:26 2022] POST /identity/v3/groups => generated 229 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:26.296351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c2bef8-e29a-47dd-bf0c-9f2ddb7837a0 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.296351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c2bef8-e29a-47dd-bf0c-9f2ddb7837a0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.296351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c2bef8-e29a-47dd-bf0c-9f2ddb7837a0 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.296351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 161/309] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:26 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:26.302601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.302601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.302601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.340734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:26.343569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:26.396698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:26.398547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae50065-e687-4bbf-874d-d41570fc63a6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:26.461491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:26.462579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.462993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.463343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] PATH_INFO: `/v3/users/82e29763ef824d3e91fcb6f4f24075b8` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.485390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 162/310] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:26.501061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] RBAC: Authorizing `identity:get_user(user_id=82e29763ef824d3e91fcb6f4f24075b8)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:26.503287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:26.505812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 149/311] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:26 2022] GET /identity/v3/users/82e29763ef824d3e91fcb6f4f24075b8 => generated 293 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:26.514582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:26.521750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:26.543111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:26.702972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0debf06d-0685-47ef-92a7-925f0f73663e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:26.707094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961344c1-729c-4c8a-92ee-78f58c80b13e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:26.769996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:26.770995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.771288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:26.771481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.771895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.772546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:26.773019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:26.773524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:26.774753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] RBAC: Adding query filter params (role.id=00bbfd499c354c379adeca6c2d8ae65e, user.id=82e29763ef824d3e91fcb6f4f24075b8, scope.system=all) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:26.775256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:26.778371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:26.793617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:26.797359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 163/312] 10.222.0.26 () {56 vars in 1308 bytes} [Sun Aug 7 00:55:26 2022] GET /identity/v3/role_assignments?user.id=82e29763ef824d3e91fcb6f4f24075b8&scope.system=all&role.id=00bbfd499c354c379adeca6c2d8ae65e => generated 226 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:26.802459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:26.803431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:26.804899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:26.805775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 150/313] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:26.811417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:26.847734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:26.962189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd9dc27-9f92-4235-8899-6553d37c4991 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:27.003249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:27.004340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:27.004827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:27.005319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:27.006187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:27.006728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:27.008424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:27.019609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 164/314] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:27.026522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:27.120295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:27.169653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c769cea6-b6a0-4a8c-b096-66b82f5d6e7a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:27.218676 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:27.219795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:27.220342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:27.220901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] PATH_INFO: `/v3/users/b582651f7d3042a9a5e612ebfdb178fc` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:27.222414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] RBAC: Authorizing `identity:get_user(user_id=b582651f7d3042a9a5e612ebfdb178fc)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:27.224581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:27.226765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 151/315] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:27 2022] GET /identity/v3/users/b582651f7d3042a9a5e612ebfdb178fc => generated 297 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:27.235050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:27.373944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:27.387630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d0676f-8ab7-4620-a0ef-367099c37511 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:27.427300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:27.428131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:27.428443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:27.428716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:27.504306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:27.505801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:27.617354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:27.622817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 165/316] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:27.631901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:27.764706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3493df55-df5d-403f-a41f-c56e46c081d9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:27.803897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe75422-3ab3-4bb4-9ae1-6e757f50c881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:27.804870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe75422-3ab3-4bb4-9ae1-6e757f50c881 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:27.805154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe75422-3ab3-4bb4-9ae1-6e757f50c881 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:27.805433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe75422-3ab3-4bb4-9ae1-6e757f50c881 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:27.806756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe75422-3ab3-4bb4-9ae1-6e757f50c881 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:27.808319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe75422-3ab3-4bb4-9ae1-6e757f50c881 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:27.810547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 152/317] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:27 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:27.816950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:27.937556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8236101d-82de-4dc2-b2d2-2d5769c52584 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:27.982047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:27.982708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:27.982993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:27.983270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:27.984043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] RBAC: Adding query filter params (role.id=00bbfd499c354c379adeca6c2d8ae65e, scope.project.id=95caef1c516c49b598639b368c2ec7ad, user.id=b582651f7d3042a9a5e612ebfdb178fc) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:27.984385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:27.986221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5ddda3-31d2-4fab-a43f-1cadf630fe70 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:27.993804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 166/318] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:27 2022] GET /identity/v3/role_assignments?user.id=b582651f7d3042a9a5e612ebfdb178fc&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=00bbfd499c354c379adeca6c2d8ae65e => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:28.633800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a7cce-0273-42d0-b396-414324e80c88 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:28.634292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a7cce-0273-42d0-b396-414324e80c88 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:28.634292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a7cce-0273-42d0-b396-414324e80c88 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:28.636362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 153/319] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:28.641477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:28.641925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:28.642018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:28.658996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d638df7b-3121-410b-a5ce-73d6406b572c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:28.659322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d638df7b-3121-410b-a5ce-73d6406b572c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:28.659584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d638df7b-3121-410b-a5ce-73d6406b572c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:28.660692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:28.661075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 154/320] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:55:28.661886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:28.666303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:28.666558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:28.666791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:28.685183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:28.686393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:28.688558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:28.714984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:28.737168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 167/321] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:28.760273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:28.768765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 155/322] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:28.789713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:28.868524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662b1897-600d-4ecb-9f94-c4a10c72176b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:28.895408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d308f25-4bcb-4962-81f2-cb4f681356e4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:28.907378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:28.908015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:28.908327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:28.908611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:28.919620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:28.924488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:28.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:28.926296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 168/323] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:28.932924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:28.936265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:28.936942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:28.938105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:28.938385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:28.939531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:28.941071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:28.943298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 156/324] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:28.949199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:28.958579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:28.981723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:29.046388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1850a22-4f18-442c-a457-8770f20efd04 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.059729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed65e1b-fb9f-41e0-b564-1a30bfc843fe None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.082535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.083153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.083477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.083706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.084420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:29.084775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:29.086174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ffdc12-1101-4f45-9a63-ecb1b850bdf5 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:29.094622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffba483-28a6-4f52-b263-e95d7bc4c678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.095002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 169/325] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:29.096204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834bad13-34e0-4a27-b097-0bc48089f6b3 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.096519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834bad13-34e0-4a27-b097-0bc48089f6b3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.096602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffba483-28a6-4f52-b263-e95d7bc4c678 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.096789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834bad13-34e0-4a27-b097-0bc48089f6b3 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.096882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffba483-28a6-4f52-b263-e95d7bc4c678 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.097138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffba483-28a6-4f52-b263-e95d7bc4c678 None admin] PATH_INFO: `/v3/groups` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.097781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffba483-28a6-4f52-b263-e95d7bc4c678 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:29.098137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 170/326] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:29.099265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffba483-28a6-4f52-b263-e95d7bc4c678 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:29.103134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834bad13-34e0-4a27-b097-0bc48089f6b3 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:29.116455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 157/327] 10.222.0.26 () {60 vars in 1134 bytes} [Sun Aug 7 00:55:28 2022] POST /identity/v3/groups => generated 226 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:55:29.117824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.118118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.118397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.138669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:29.139922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:29.212342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:29.224141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 158/328] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:29.249140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:29.250570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834bad13-34e0-4a27-b097-0bc48089f6b3 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:29.306063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.308292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.308292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.308292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] PATH_INFO: `/v3/users/8a19427696d94a4fa4987bb65cfd9e8e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.309362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] RBAC: Authorizing `identity:get_user(user_id=8a19427696d94a4fa4987bb65cfd9e8e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:29.310889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:29.316432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 171/329] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:29 2022] GET /identity/v3/users/8a19427696d94a4fa4987bb65cfd9e8e => generated 293 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:29.321417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:29.398616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1ef0a6-5e5e-428f-be23-a844a5abcf38 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.440893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.441617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.441931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.442210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.455461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:29.460901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:29.461868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:29.462713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:29.464834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 159/330] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:29.467871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666e281-d626-4892-b9ae-85c779efe61d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.470178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:29.521917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.521917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.521917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.521917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.523237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] RBAC: Adding query filter params (role.id=e24199f1fbf64edfbef577416cf3c4b2, user.id=8a19427696d94a4fa4987bb65cfd9e8e, scope.system=all) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:29.523237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:29.524686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:29.547021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 172/331] 10.222.0.26 () {56 vars in 1308 bytes} [Sun Aug 7 00:55:29 2022] GET /identity/v3/role_assignments?user.id=8a19427696d94a4fa4987bb65cfd9e8e&scope.system=all&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 510 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:29.551446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:29.619735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64394274-afdd-4b59-8954-ba5fdfbff7c7 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.674990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.675736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.675989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.676169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.676913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:29.677211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:29.678658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:29.690062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 160/332] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:29.695658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:29.740049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:29.826623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53fa04d-a22c-4cd2-8342-d0f88a9dd220 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:29.877961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfdc06d-4348-4682-91bc-2e4f929ab3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:29.878871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfdc06d-4348-4682-91bc-2e4f929ab3f9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:29.879330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfdc06d-4348-4682-91bc-2e4f929ab3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:29.879731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfdc06d-4348-4682-91bc-2e4f929ab3f9 None admin] PATH_INFO: `/v3/users/82e29763ef824d3e91fcb6f4f24075b8` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:29.881169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfdc06d-4348-4682-91bc-2e4f929ab3f9 None admin] RBAC: Authorizing `identity:get_user(user_id=82e29763ef824d3e91fcb6f4f24075b8)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:29.883807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfdc06d-4348-4682-91bc-2e4f929ab3f9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:29.886182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 173/333] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:29 2022] GET /identity/v3/users/82e29763ef824d3e91fcb6f4f24075b8 => generated 293 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:29.891816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:30.008444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edbddce-de62-47fc-a3a1-59bb4f016af9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:30.031550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:30.058026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c294d-4e91-4feb-bda7-ec2ad3aa5e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:30.058964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c294d-4e91-4feb-bda7-ec2ad3aa5e2f None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:30.059401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c294d-4e91-4feb-bda7-ec2ad3aa5e2f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:30.059823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c294d-4e91-4feb-bda7-ec2ad3aa5e2f None admin] PATH_INFO: `/v3/system/users/82e29763ef824d3e91fcb6f4f24075b8/roles/00bbfd499c354c379adeca6c2d8ae65e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:30.061519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99c294d-4e91-4feb-bda7-ec2ad3aa5e2f None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=82e29763ef824d3e91fcb6f4f24075b8, role_id=00bbfd499c354c379adeca6c2d8ae65e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:30.063244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99c294d-4e91-4feb-bda7-ec2ad3aa5e2f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:30.073608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 161/334] 10.222.0.26 () {58 vars in 1334 bytes} [Sun Aug 7 00:55:29 2022] PUT /identity/v3/system/users/82e29763ef824d3e91fcb6f4f24075b8/roles/00bbfd499c354c379adeca6c2d8ae65e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:32.002246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd0e17-6969-40a7-b4ec-7631d89f5925 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.002817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd0e17-6969-40a7-b4ec-7631d89f5925 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.003140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd0e17-6969-40a7-b4ec-7631d89f5925 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.005466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 174/335] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:32.016255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.016996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.017503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.062061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:32.064190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:32.227551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 162/336] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:32.275276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd0e17-6969-40a7-b4ec-7631d89f5925 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:32.428329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:32.435113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd0e17-6969-40a7-b4ec-7631d89f5925 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:32.477838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292267d-bc70-41e9-b19f-82090491b8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:32.479011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292267d-bc70-41e9-b19f-82090491b8db None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.479292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292267d-bc70-41e9-b19f-82090491b8db None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.479564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292267d-bc70-41e9-b19f-82090491b8db None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.498782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7292267d-bc70-41e9-b19f-82090491b8db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-7292267d-bc70-41e9-b19f-82090491b8db None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:32.505056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:32.506053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:32.506873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 175/337] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:32.512964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:32.525858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192f7b86-242b-4c24-99f4-1d41eb0fc6f7 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.526039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192f7b86-242b-4c24-99f4-1d41eb0fc6f7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.526323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192f7b86-242b-4c24-99f4-1d41eb0fc6f7 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.530389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 176/338] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:32 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:32.536300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83f053e-7f83-4650-baf0-f28082b943d1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.536582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83f053e-7f83-4650-baf0-f28082b943d1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.536816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83f053e-7f83-4650-baf0-f28082b943d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.577374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e83f053e-7f83-4650-baf0-f28082b943d1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:32.578597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e83f053e-7f83-4650-baf0-f28082b943d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:32.687153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 177/339] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:32.688720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977d6a32-75aa-4c08-9eab-8e70a64eb49b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.689519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977d6a32-75aa-4c08-9eab-8e70a64eb49b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.689970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977d6a32-75aa-4c08-9eab-8e70a64eb49b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.691840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 178/340] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:32.695526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664ff436-a4ff-4b9e-9410-2ea560f09868 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:32.699223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.699814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.700310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.720612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:32.723133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:32.724813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:32.761991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:32.762630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:32.762912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:32.763185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:32.763874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:32.764176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:32.765735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:32.787979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 163/341] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:32.789212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:32.820387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 179/342] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:32.821652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:32.876680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:32.960086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c168b1ad-0e3a-48d3-850b-c633ea90a94e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.000911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad61191-546f-4f39-8d43-02f5385ec9bc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.002729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.004053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.004540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.004985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.088447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.089717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.090153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.090571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.116147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:33.122906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:33.124027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:33.125030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:33.130438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 164/343] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:33.134145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.148759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:33.153362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:33.201730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:33.216650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:33.275404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 180/344] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:33.277689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.313725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4428d016-9fb3-4827-a2c0-456bd1c0b6b6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.367889 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.368512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.369134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.369134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.384497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:33.397347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec985af-ab7e-45fe-b869-cd2cbfe5cd8b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:33.398667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:33.399729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:33.401269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 165/345] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:33.402272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.428622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:33.443282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.444088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.444423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.444704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.445381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:33.445689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:33.448375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:33.458016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 181/346] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:33.459331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.559940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:33.576954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7761fa28-ec35-447f-81b8-9e9b154ce6cc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.605763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a871dcb9-6895-4516-993b-ef87a21214fb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.641748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.642071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.642339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.642612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.643789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:33.645283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:33.646976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 166/347] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:33.647768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.651946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.653045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.653625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.654096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.654995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:33.655496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:33.657197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:33.666985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 182/348] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:33.669500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.722632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:33.772000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1f0298-2df8-47ef-9801-fd1d779d759f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.788281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0ac141-81f9-4b2e-86cc-0d1117ed21ff None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.811978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5034f49c-5058-47d7-b9ea-f8b9d0b9ac38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.812395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5034f49c-5058-47d7-b9ea-f8b9d0b9ac38 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.812681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5034f49c-5058-47d7-b9ea-f8b9d0b9ac38 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.812957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5034f49c-5058-47d7-b9ea-f8b9d0b9ac38 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.829317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:33.830364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:33.830937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:33.831493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:33.839049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:33.841043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:33.843278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 183/349] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:33.844657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:33.850531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:33.896985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5034f49c-5058-47d7-b9ea-f8b9d0b9ac38 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:33.898520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5034f49c-5058-47d7-b9ea-f8b9d0b9ac38 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:33.971533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2817c3-aa3d-4500-9e28-21ac6b3d1f3d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:33.983978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:34.020944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf521c5-9752-4381-b5b9-7c75c6bf515b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:34.021590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf521c5-9752-4381-b5b9-7c75c6bf515b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.024286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf521c5-9752-4381-b5b9-7c75c6bf515b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.024286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf521c5-9752-4381-b5b9-7c75c6bf515b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.049153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 167/350] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:34.049958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646f5a09-37d4-4163-9315-ae6394e6128a None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.050244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646f5a09-37d4-4163-9315-ae6394e6128a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.050516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646f5a09-37d4-4163-9315-ae6394e6128a None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.052044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 168/351] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:34.052527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646f5a09-37d4-4163-9315-ae6394e6128a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:34.112904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf521c5-9752-4381-b5b9-7c75c6bf515b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:34.114496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf521c5-9752-4381-b5b9-7c75c6bf515b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:34.132141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:34.178921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646f5a09-37d4-4163-9315-ae6394e6128a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:34.219618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:34.220479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.221460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.221460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.221758 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:34.222069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:34.224065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c774dc42-450c-4da8-9581-7d68c1e1472d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:34.231491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 169/352] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:34 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 260 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:34.232741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1671b357-6ede-432e-9ba0-c3c3bee0f6b2 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.233040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1671b357-6ede-432e-9ba0-c3c3bee0f6b2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.233665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1671b357-6ede-432e-9ba0-c3c3bee0f6b2 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.235132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 170/353] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:34.235746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd43cd-6e1b-4be2-82dd-329d1028b659 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.236129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd43cd-6e1b-4be2-82dd-329d1028b659 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.236528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbd43cd-6e1b-4be2-82dd-329d1028b659 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.238011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 171/354] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:34.238638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd43cd-6e1b-4be2-82dd-329d1028b659 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:34.249372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 184/355] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:34.251840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86048ee1-441e-42a0-98de-b8ffc16ecfe3 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.252119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86048ee1-441e-42a0-98de-b8ffc16ecfe3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.252461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86048ee1-441e-42a0-98de-b8ffc16ecfe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.266512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:34.274381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-86048ee1-441e-42a0-98de-b8ffc16ecfe3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:34.278266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-86048ee1-441e-42a0-98de-b8ffc16ecfe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:34.378527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 185/356] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:34.384336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.384336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.384336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.406101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbd43cd-6e1b-4be2-82dd-329d1028b659 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:34.419000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:34.420967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:34.439419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:34.453910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9a244e-3c52-4849-8508-bae3f11b36b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:34.455236 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9a244e-3c52-4849-8508-bae3f11b36b1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.455883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9a244e-3c52-4849-8508-bae3f11b36b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.456269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9a244e-3c52-4849-8508-bae3f11b36b1 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.459774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9a244e-3c52-4849-8508-bae3f11b36b1 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:34.462439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9a244e-3c52-4849-8508-bae3f11b36b1 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:34.465840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 172/357] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:34 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:34.468046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.469466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.469466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.503892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:34.505794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:34.535669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 186/358] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:34.536690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:34.634240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 173/359] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:34.635261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:34.695442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:34.749483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f8bc49-6edc-48f2-bdc3-9a05a9020080 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:34.816263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aad36a-ec92-4045-acd2-3607898d80b8 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:34.844655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:34.845314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.845603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.845889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.847479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:34.849316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:34.851246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 187/360] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:34 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:34.851846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:34.872491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:34.890195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:34.890864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:34.891146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:34.891417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:34.911531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:34.920353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:34.925803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 174/361] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:34.927424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.050864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab6e407-e301-4390-bb11-55661c435799 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.121858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.124223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.124223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.124551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.128951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.129544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.133149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 188/362] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:34 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.134275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.139144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:35.146623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8384df81-4a27-4e55-bdf4-0b128b26c029 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.194280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.195328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.195847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.196223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.209993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:35.216413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:35.217334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:35.218130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 175/363] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:35.218850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.288597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585dd22-f8d5-4cb1-9f9a-13161a6d5fe1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.298494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:35.338140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.338716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.339025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.339301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.351576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:35.357444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:35.358348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 189/364] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:35.359529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.393545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6bddbf-8d87-4d80-b1a6-e678537d31f9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.439416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.440453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.440956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.441377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:35.441508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.442570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.444073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.446197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 176/365] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.447183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.489614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a0c9ae-496a-4f68-b48b-70aa15071d19 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.537625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.538334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.538879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.539185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.539875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:35.540179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.542976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.553932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 190/366] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.554391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.611199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec13d9-000f-4f99-860b-c1bec8524b36 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.625704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:35.651523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.652310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.652698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.653089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.653927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:35.654307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.655851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.662037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 177/367] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 260 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.662834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.706651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002504c9-fd60-455e-b02d-ded194802a5c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.736806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:35.754208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.754864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.755195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.755475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.756162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:35.756503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.757950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.770902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 191/368] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.771295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.813697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45195512-16e9-4c34-87fb-cbcaad347d04 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.865122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.866112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.866598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.867077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.867983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] RBAC: Adding query filter params (name=reader) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:35.868542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.871019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.889087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 178/369] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.889992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:35.921847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1fa5e-a89f-499a-94c7-4c4ef7b284ec None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:35.949503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:35.972052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:35.973265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:35.973785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:35.974446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:35.975859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=95caef1c516c49b598639b368c2ec7ad) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:35.976453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:35.979831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:35.998045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 192/370] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 260 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:35.999705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:36.126931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be6c4b6-e0e7-43e0-b30f-f0fd53c848fe None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:36.174002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:36.254214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:36.255312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.255819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.256378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.318217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49da7eef-12ce-4aa8-bb1f-3eda7e051f69 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:36.371815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5b4e64-b13b-43a1-9a8f-8c7f40e121e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:36.373383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b4e64-b13b-43a1-9a8f-8c7f40e121e7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.373873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b4e64-b13b-43a1-9a8f-8c7f40e121e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.374152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5b4e64-b13b-43a1-9a8f-8c7f40e121e7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.418200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:36.419596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:36.503354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:36.544885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5b4e64-b13b-43a1-9a8f-8c7f40e121e7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:36.546379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5b4e64-b13b-43a1-9a8f-8c7f40e121e7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:36.570896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 179/371] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 681 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:36.571783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:36.615482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:36.678790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 193/372] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 681 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:36.680479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eb7fac-2dc8-4484-a87a-3d611362c65b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.680772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eb7fac-2dc8-4484-a87a-3d611362c65b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.681050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eb7fac-2dc8-4484-a87a-3d611362c65b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.683291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 194/373] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:36.684046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eb7fac-2dc8-4484-a87a-3d611362c65b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:36.714294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7530883-211a-469e-bc32-a0f7cdb11d92 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:36.762296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:36.763065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.763638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.763921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] PATH_INFO: `/v3/users/82e29763ef824d3e91fcb6f4f24075b8` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.765400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] RBAC: Authorizing `identity:get_user(user_id=82e29763ef824d3e91fcb6f4f24075b8)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:36.767807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:36.769771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 180/374] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:55:36 2022] GET /identity/v3/users/82e29763ef824d3e91fcb6f4f24075b8 => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:36.770562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:36.783998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:36.812464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eb7fac-2dc8-4484-a87a-3d611362c65b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:36.852969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a85034-3655-4221-8de5-82d9356b1843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:36.854211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a85034-3655-4221-8de5-82d9356b1843 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.854522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a85034-3655-4221-8de5-82d9356b1843 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.854897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a85034-3655-4221-8de5-82d9356b1843 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.856324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a85034-3655-4221-8de5-82d9356b1843 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:36.857802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a85034-3655-4221-8de5-82d9356b1843 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:36.864355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 195/375] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:36 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:36.865681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.865969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.866246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.886249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:36.887561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:36.905025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:36.911263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0933049-598c-44ca-96dd-326680fb5a6b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:36.950509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:36.951346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:36.951637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:36.951917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] PATH_INFO: `/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:36.953116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] RBAC: Authorizing `identity:get_project(project_id=ae3747bdb31f45b9bc97b22fb3bfe021)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:36.954826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:36.957045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 181/376] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:36 2022] GET /identity/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021 => generated 344 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:36.957880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:36.969191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 196/377] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:36.970336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.075294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:37.080107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9a018e-580e-4646-98f8-c83ea84e0657 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.105784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dcc8cc-143c-4200-8c37-29fe0daadd79 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.117280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.118253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.118767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.119795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.121216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.122904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.124764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 197/378] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:36 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.125810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.149420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.150049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.150860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.151373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.152151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] RBAC: Adding query filter params (role.id=00bbfd499c354c379adeca6c2d8ae65e, user.id=82e29763ef824d3e91fcb6f4f24075b8, scope.system=all) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:37.152512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.152628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:37.154459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.172663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 182/379] 10.222.0.26 () {56 vars in 1308 bytes} [Sun Aug 7 00:55:36 2022] GET /identity/v3/role_assignments?user.id=82e29763ef824d3e91fcb6f4f24075b8&scope.system=all&role.id=00bbfd499c354c379adeca6c2d8ae65e => generated 510 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.173681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.272214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d861e1-b81d-4d96-b217-046af02e586a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.317982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.319027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.319523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.319991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] PATH_INFO: `/v3/groups/40190e63afe24695970c19539106579f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.326739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] RBAC: Authorizing `identity:get_group(group_id=40190e63afe24695970c19539106579f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.336268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.336268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 198/380] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/groups/40190e63afe24695970c19539106579f => generated 229 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.336695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.404592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf2ec26-63b9-4117-95c5-19e7edd44913 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.437155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:37.444406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.444969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.445253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.445529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.459622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:37.465176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:37.466085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:37.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 183/381] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:37.467562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.489749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a836a7b-c95d-419a-be73-e9e854e00578 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.493376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:37.528158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.529121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.529618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.530090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.531064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=95caef1c516c49b598639b368c2ec7ad) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:37.531592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.533262 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.540663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 199/382] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 260 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.542407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.627133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278d8a97-63ed-4c8d-a9a5-0dd75cdbb791 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.665683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d66d16-f7b6-483f-9679-2e61064dac31 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.698189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.698823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.699137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.699422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.700169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] RBAC: Adding query filter params (group.id=40190e63afe24695970c19539106579f, role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=ae3747bdb31f45b9bc97b22fb3bfe021) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:37.700512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.701992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.706941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.707897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.708430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.708908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.709825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:37.710360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.712067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.713921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 184/383] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/role_assignments?group.id=40190e63afe24695970c19539106579f&scope.project.id=ae3747bdb31f45b9bc97b22fb3bfe021&role.id=b0ec0093717c472da3640ec7121208d4 => generated 260 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.724653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 200/384] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.735598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.821146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:37.880265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465eb115-8e48-4a06-acad-df27158bab58 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:37.925148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:37.925826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:37.926109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:37.926388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:37.927663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:37.929179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:37.931485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 185/385] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:37.937424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:37.956633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:38.075516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7ff41-2d7e-47af-a297-caf1174d755a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:38.114860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb308b8-6c56-401c-abf8-e8fe83b5c0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:38.115511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb308b8-6c56-401c-abf8-e8fe83b5c0ae None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.115788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb308b8-6c56-401c-abf8-e8fe83b5c0ae None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.116066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb308b8-6c56-401c-abf8-e8fe83b5c0ae None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.117268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb308b8-6c56-401c-abf8-e8fe83b5c0ae None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:38.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb308b8-6c56-401c-abf8-e8fe83b5c0ae None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:38.121416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 201/386] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:37 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:38.127533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:38.245892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04729f16-a8e7-41e5-aed1-3c0da895170a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:38.272570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:38.291802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4677f79b-ad23-4219-8721-0cd1e01855a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:38.292617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677f79b-ad23-4219-8721-0cd1e01855a0 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.292922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677f79b-ad23-4219-8721-0cd1e01855a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.293202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4677f79b-ad23-4219-8721-0cd1e01855a0 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.295182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4677f79b-ad23-4219-8721-0cd1e01855a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=6fe07ee094f944898ef10a16ce01c7c3, group_id=fb2509f84bdd4fe68f8d8c125cbdb891, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:38.296781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4677f79b-ad23-4219-8721-0cd1e01855a0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:38.316282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 186/387] 10.222.0.26 () {58 vars in 1442 bytes} [Sun Aug 7 00:55:38 2022] PUT /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:38.323083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14135e2-eb22-4ed9-afdd-04fb39809269 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.323602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14135e2-eb22-4ed9-afdd-04fb39809269 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.323711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14135e2-eb22-4ed9-afdd-04fb39809269 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.325287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 202/388] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:38.340322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.340614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.340916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.363863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:38.365176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:38.378909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1edcca6-05c8-43ac-bbc4-f95b83be16d4 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.379245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1edcca6-05c8-43ac-bbc4-f95b83be16d4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.379471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1edcca6-05c8-43ac-bbc4-f95b83be16d4 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.381096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 203/389] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:38.386910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eba170-5790-4660-b339-2146de411f05 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.387219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eba170-5790-4660-b339-2146de411f05 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.387455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eba170-5790-4660-b339-2146de411f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.423825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-35eba170-5790-4660-b339-2146de411f05 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:38.425122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-35eba170-5790-4660-b339-2146de411f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:38.482100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 187/390] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:38.519204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:38.529844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 204/391] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:38.559238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eba170-5790-4660-b339-2146de411f05 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:38.562555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:38.690519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82c652c-d5d9-47c1-9e90-c634b3a97790 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:38.710468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:38.715037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eba170-5790-4660-b339-2146de411f05 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:38.736716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:38.737365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.738134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.738408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.750311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:38.755622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:38.756660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:38.757468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:38.758086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 188/392] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:38.765615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:38.799292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:38.799945 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.800248 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.800536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.829386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:38.834565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:38.835466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:38.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:38.840659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 205/393] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:38.845397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:38.890572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad5a70f-ad52-4b0f-ad2c-7abec5373fec None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:38.958365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:38.958904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:38.959183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:38.959451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:38.960121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:38.960443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:38.961827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:38.972195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 189/394] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:38.977758 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:38.989990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:39.041890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:39.079136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a255de17-a002-4330-872c-0d236b29e446 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.119727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d84d37-d2b3-4343-82ff-88ed33afdf37 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.150852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.152014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.152550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.153142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.154072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] RBAC: Adding query filter params (name=member) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:39.154591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.156464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.167042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.167723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.168104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.168317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.168800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 206/395] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:39.172273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.172273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.173335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 190/396] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:38 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:39.175368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:39.181356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:39.313819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3f6410-2ca9-48c6-8656-f9cc3249288b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.344857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:39.359929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.360598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.360880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.361147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.362259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.363836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.370018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 207/397] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:39 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:39.371273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:39.418380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df50c9f-a6c1-4869-a806-c7a034a42557 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.425254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:39.475547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b96b98-fdd4-4d54-a5ba-e707494a497f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.513157 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.514045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.514330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.514603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.515758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.517656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.523814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 208/398] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:39 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:39.525816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:39.573594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.574227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.574458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.574733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.580986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.582459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.584157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 191/399] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:39 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 403 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:39.593198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:39.636839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3851c5b-3c98-429b-9e92-64ae95cfcf29 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.640673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:39.678234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d8f58d-961b-4917-b409-61223bb98f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.679020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d8f58d-961b-4917-b409-61223bb98f1c None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.679742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d8f58d-961b-4917-b409-61223bb98f1c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.680084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d8f58d-961b-4917-b409-61223bb98f1c None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.683308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d8f58d-961b-4917-b409-61223bb98f1c None admin] RBAC: Authorizing `identity:create_grant(project_id=95caef1c516c49b598639b368c2ec7ad, group_id=fb2509f84bdd4fe68f8d8c125cbdb891, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.685404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d8f58d-961b-4917-b409-61223bb98f1c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.723429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 209/400] 10.222.0.26 () {58 vars in 1442 bytes} [Sun Aug 7 00:55:39 2022] PUT /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:39.774259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c146cc1-d554-41e5-974a-01e81049de02 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:39.816735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f04824-f973-4831-be7f-e2a8eccec909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:39.817563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f04824-f973-4831-be7f-e2a8eccec909 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:39.817853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f04824-f973-4831-be7f-e2a8eccec909 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:39.818132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f04824-f973-4831-be7f-e2a8eccec909 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:39.820637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f04824-f973-4831-be7f-e2a8eccec909 None admin] RBAC: Authorizing `identity:create_grant(project_id=6fe07ee094f944898ef10a16ce01c7c3, group_id=fb2509f84bdd4fe68f8d8c125cbdb891, role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:39.822720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f04824-f973-4831-be7f-e2a8eccec909 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:39.843243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 192/401] 10.222.0.26 () {58 vars in 1442 bytes} [Sun Aug 7 00:55:39 2022] PUT /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:40.292968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c2307-bb8b-49af-adf9-b7141e2d5390 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.293256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c2307-bb8b-49af-adf9-b7141e2d5390 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.293516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c2307-bb8b-49af-adf9-b7141e2d5390 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.295167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 210/402] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:40.301648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.301648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.301648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.309744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec62efb5-596f-46c5-9048-03df5dd7d1fa None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.310025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec62efb5-596f-46c5-9048-03df5dd7d1fa None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.310304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec62efb5-596f-46c5-9048-03df5dd7d1fa None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.312899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 211/403] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:40.320708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.320990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.321264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.337692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:40.338924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:40.356874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:40.358080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:40.383662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:40.461470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 193/404] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:40.475328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 212/405] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:40.488220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:40.503126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:40.604576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:40.679501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe58b4f2-b78e-4981-8d2d-c01e3528210b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:40.703377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62d62cd-f97a-485b-a908-864d4c71ec51 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:40.756360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:40.757012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.757286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.757549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.777590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:40.778354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:40.779237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.779737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.780260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:40.782484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:40.783360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:40.784186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:40.788472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 194/406] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:40.789727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:40.818370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:40.819302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:40.836267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:40.837313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:40.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:40.840591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 213/407] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:40.846439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:40.902327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc955a90-b7a0-4b8d-b5b9-97380548cf78 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:40.941205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:40.941824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:40.942359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:40.942655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:40.943328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:40.944124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:40.945731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:40.962063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 195/408] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:40.963423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.068622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:41.085253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534eb0e-128e-4d29-aa5e-98e9b046f33d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.106674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab6e763-7ba9-4d06-af60-9be5aafe2816 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.122479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635d7cce-15da-4d10-83fc-a7f0c15ddb24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.123215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635d7cce-15da-4d10-83fc-a7f0c15ddb24 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.123621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635d7cce-15da-4d10-83fc-a7f0c15ddb24 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.123960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635d7cce-15da-4d10-83fc-a7f0c15ddb24 None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.125226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635d7cce-15da-4d10-83fc-a7f0c15ddb24 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.126775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635d7cce-15da-4d10-83fc-a7f0c15ddb24 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.127234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:41.134412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 196/409] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:40 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:41.136553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68c02c6-df31-4ee0-b515-1e26ced63cfb None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.137014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68c02c6-df31-4ee0-b515-1e26ced63cfb None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.137431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68c02c6-df31-4ee0-b515-1e26ced63cfb None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.139381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 197/410] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:41.140415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68c02c6-df31-4ee0-b515-1e26ced63cfb None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.155007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.155544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.155829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.156106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.156812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:41.157120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.158779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a5390d-4747-40f0-83bb-1cdb117a762e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.168374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 214/411] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:41.169858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.170159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.170422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.190370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:41.191629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:41.252792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68c02c6-df31-4ee0-b515-1e26ced63cfb None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.268187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 215/412] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:41.277216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.298440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.299852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.301143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.301613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.305105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.307137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.313882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 198/413] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:41 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:41.314759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.338120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:41.419161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899cf26a-e1db-40a0-a4cc-89ac4fbe2021 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.428693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee451609-f5a9-4aac-aad8-c4d6c7487b45 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.440677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:41.465109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.465752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.466029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.466301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.466587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.467559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.468391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.469445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] PATH_INFO: `/v3/groups/40190e63afe24695970c19539106579f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.471012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] RBAC: Authorizing `identity:get_group(group_id=40190e63afe24695970c19539106579f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.472991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.475479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 216/414] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:41 2022] GET /identity/v3/groups/40190e63afe24695970c19539106579f => generated 229 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:41.476539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.478002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:41.483731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:41.484699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:41.485547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 199/415] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:41.486203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.601091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:41.613495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4695880b-0337-48eb-a7c2-8817428130b1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.622540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eaee94-de62-4055-acfd-32559dd3f993 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.653445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.654505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.654987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.655482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.657929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.658490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] RBAC: Authorizing `identity:create_grant(project_id=95caef1c516c49b598639b368c2ec7ad, group_id=fb2509f84bdd4fe68f8d8c125cbdb891, role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.658599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.658829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.659099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] PATH_INFO: `/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.660309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] RBAC: Authorizing `identity:get_project(project_id=ae3747bdb31f45b9bc97b22fb3bfe021)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.661079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.661808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.664060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 200/416] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:41 2022] GET /identity/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021 => generated 344 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:41.664831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.678401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 217/417] 10.222.0.26 () {58 vars in 1442 bytes} [Sun Aug 7 00:55:41 2022] PUT /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad/groups/fb2509f84bdd4fe68f8d8c125cbdb891/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:41.680013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:41.768874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:41.868216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a5a6ef-be9d-4ebd-9912-dede03a9d4a6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.902967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001080e9-6b11-4207-8026-61e8780483f4 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:41.910986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.912066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.912612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.913139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] PATH_INFO: `/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021/groups/40190e63afe24695970c19539106579f/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.915591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae3747bdb31f45b9bc97b22fb3bfe021, group_id=40190e63afe24695970c19539106579f, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.917517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.935986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 218/418] 10.222.0.26 () {58 vars in 1442 bytes} [Sun Aug 7 00:55:41 2022] PUT /identity/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021/groups/40190e63afe24695970c19539106579f/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:55:41.941426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:41.941426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:41.941737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:41.941818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:41.942507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:41.942809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:41.944226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6521822-802d-452d-aa3b-61f7dc08ddf0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:41.956786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 201/419] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:41.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.095107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.107328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41be180f-2a57-480b-95b0-36f3c458fb92 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:42.158116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:42.158745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.159022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.159292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.198579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92088e2d-3fa1-4c09-90c6-fe14c2cff26b None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.198871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92088e2d-3fa1-4c09-90c6-fe14c2cff26b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.199059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92088e2d-3fa1-4c09-90c6-fe14c2cff26b None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.200778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 202/420] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:42.206025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3b9c44-e20a-47be-ac6f-24ab9847a036 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.206324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3b9c44-e20a-47be-ac6f-24ab9847a036 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.206746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3b9c44-e20a-47be-ac6f-24ab9847a036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.245477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ca3b9c44-e20a-47be-ac6f-24ab9847a036 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:42.246740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ca3b9c44-e20a-47be-ac6f-24ab9847a036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:42.258699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:42.261070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:42.264327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.371043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 203/421] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:42.372604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c775bb-d0c0-4d4a-8c53-83d1b95e6dec None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.372892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c775bb-d0c0-4d4a-8c53-83d1b95e6dec None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.373174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c775bb-d0c0-4d4a-8c53-83d1b95e6dec None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.374875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 204/422] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:42.380581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.380911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.381139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.402846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:42.404437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:42.405542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.419313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 219/423] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:42.420228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.499315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 205/424] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:42.500117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.529469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b9a231-23ac-46f3-a3d2-4eb2d8f7b688 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:42.568496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:42.569185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.569464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.569734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.581114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:42.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:42.586974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:42.588836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 220/425] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:42.590561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.604282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.641206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5872a15c-bd55-4bd2-a014-0ffad45148d5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:42.652846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.684052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:42.684903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.685283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.685657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.686983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:42.688640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:42.690770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 206/426] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:42.691774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.725510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3db18f0-ea44-4040-9036-af4861e4e0c0 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:42.766840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:42.767619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.768041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.768459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.780983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:42.786762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:42.787950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:42.788995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:42.791992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 221/427] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:42.791992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.841998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a611d48-faec-4d00-a809-bd485e561fa6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:42.890558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:42.891697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.892130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.892577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.893715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] RBAC: Adding query filter params (name=member) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:42.895101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:42.896746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:42.904539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.910249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 207/428] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:42.911877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:42.935982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2301e57e-ebcc-437a-bd05-7fb014cfc0c5 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:42.954947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:42.981314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:42.982260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:42.982763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:42.983267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:42.984926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:42.987511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:42.990630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 222/429] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:42.992147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.038819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a321d9-3189-46b3-a4f4-1f5effc338ec None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.077968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.078710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.078998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.079273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.079965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:43.080409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.082301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.097804 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 208/430] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:43.099102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.123602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bda778-4fd4-4212-8163-cf1832f440a5 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.172492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.173872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.174388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.175112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:43.175565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.223732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5b8fc-473b-448c-b535-73329ad76534 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.250296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:43.262860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.263485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.263767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.264042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.264974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:43.265218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.266872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.274433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 209/431] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 611 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:43.278277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.287395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.290252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.405083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b236bf-a844-41bc-a4e4-ed8a65180ef7 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.437030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:43.448778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.449438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.449721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.450056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.452389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 223/432] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:43.462355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.580421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.583528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.595187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:43.625210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcd034-2838-479e-8472-0a92b034f8dc None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.672922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.673937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.674286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.674567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.676015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.678206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.680809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 224/433] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:43.688284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.746028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 210/434] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:43.751752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.785925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:43.846229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f05256f-78ec-43f5-9055-08af088b8d27 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.903460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e470d-0ea8-445c-a2ac-3263be60ad9a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:43.906776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.908205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.908786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.910343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.911925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.913711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.917412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 225/435] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:43.924668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:43.953102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:43.957263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:43.957921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:43.958195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:43.958468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] PATH_INFO: `/v3/projects/6fe07ee094f944898ef10a16ce01c7c3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:43.959669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] RBAC: Authorizing `identity:get_project(project_id=6fe07ee094f944898ef10a16ce01c7c3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:43.962174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:43.967835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 211/436] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/projects/6fe07ee094f944898ef10a16ce01c7c3 => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:43.974059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.083289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eb4ca9-e3c7-4964-8e36-921192a0bb7e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.109963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:44.121762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633dcd5d-4c70-4c55-bd9f-e0121cec0e52 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.137576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.138831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.139323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.140397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.141644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=e24199f1fbf64edfbef577416cf3c4b2, scope.project.id=95caef1c516c49b598639b368c2ec7ad) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:44.142192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:44.143908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:44.153343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 226/437] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=e24199f1fbf64edfbef577416cf3c4b2 => generated 611 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:44.180697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbbc51c-8c31-4556-8a8e-f7fa3dcbcd5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.181859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbbc51c-8c31-4556-8a8e-f7fa3dcbcd5d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.182365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbbc51c-8c31-4556-8a8e-f7fa3dcbcd5d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.182865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbbc51c-8c31-4556-8a8e-f7fa3dcbcd5d None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.184520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbbc51c-8c31-4556-8a8e-f7fa3dcbcd5d None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:44.186301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbbc51c-8c31-4556-8a8e-f7fa3dcbcd5d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:44.188542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 212/438] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:43 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:44.194631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.278022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6861c7-c063-425b-802e-3841ef3bca3e None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.278346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6861c7-c063-425b-802e-3841ef3bca3e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.279551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6861c7-c063-425b-802e-3841ef3bca3e None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.281131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 213/439] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:44.287397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.287724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.287946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.309992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:44.313038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:44.314448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:44.337142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57092538-9dcc-4caa-b607-4a404ba1a8f8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.386092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.386619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.386903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.387185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.387954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=6fe07ee094f944898ef10a16ce01c7c3) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:44.388358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:44.389823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8201a9f6-4e9c-4dae-bb76-19bf6ea5bcef None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:44.399494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 227/440] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=6fe07ee094f944898ef10a16ce01c7c3&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 611 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:44.400973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22932e53-e153-4ef1-9e77-60b45052e7e0 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.401261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22932e53-e153-4ef1-9e77-60b45052e7e0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.401541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22932e53-e153-4ef1-9e77-60b45052e7e0 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.402861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 228/441] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:44.409323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.409673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.409885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.416323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 214/442] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:44.433868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:44.435071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:44.440151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.522142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 229/443] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:44.547444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.561770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:44.573299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f807027-f3ac-40c4-abcc-33bc39f29fe4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.619278 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.619925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.620200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.620510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.633914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:44.637606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:44.639341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:44.640545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:44.641344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 215/444] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:44.648877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.668779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eeb70c-7ccc-4ed6-870c-8749869961bf None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.714502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.715147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.715424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.715714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.729805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:44.739852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:44.740865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:44.741667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 230/445] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:44.748502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.785764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1499755-7747-4847-a721-2999a644a9ea None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.831183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.832151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.832484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.832771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.833482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:44.833792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:44.835241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:44.849411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 216/446] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:44.855254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:44.878645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593d96a1-7ca7-4779-bbb4-86db63315eb0 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:44.881047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:44.928603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:44.929147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:44.929687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:44.929966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:44.930655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:44.930960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:44.932978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:44.934680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:44.945039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 231/447] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:44.951390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.016039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3a840-301c-435d-b556-ac961a174f63 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.060423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.061042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.061353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.061605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.079123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d65c4-4b44-4320-91fc-aa97a978de1e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.115052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.115669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.115944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.116263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.139262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.140788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.156269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:45.180179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:45.187574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.189622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.247167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 217/448] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:45.252927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.297996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 232/449] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 07 00:55:45.304047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.358571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb65b34-7419-4f39-9fff-161d271b9b41 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.373573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:45.395415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.396184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.396493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.396762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] PATH_INFO: `/v3/projects/95caef1c516c49b598639b368c2ec7ad` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.397994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] RBAC: Authorizing `identity:get_project(project_id=95caef1c516c49b598639b368c2ec7ad)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.399512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.401952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 218/450] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:45 2022] GET /identity/v3/projects/95caef1c516c49b598639b368c2ec7ad => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:45.405881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:45.407770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.412367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba372cd-95fb-42b5-9eb8-3ac27ebcbdd9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.450317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.450317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.450317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.450763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] PATH_INFO: `/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.451640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] RBAC: Authorizing `identity:get_project(project_id=ae3747bdb31f45b9bc97b22fb3bfe021)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.453163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.454816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 233/451] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:55:45 2022] GET /identity/v3/projects/ae3747bdb31f45b9bc97b22fb3bfe021 => generated 344 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:45.460825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.513085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21bf2e8-517f-4676-b51c-203585cbaf18 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.548592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.549230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.549509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.549785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] PATH_INFO: `/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.550919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] RBAC: Authorizing `identity:get_group(group_id=fb2509f84bdd4fe68f8d8c125cbdb891)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.552426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.554029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 219/452] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:45 2022] GET /identity/v3/groups/fb2509f84bdd4fe68f8d8c125cbdb891 => generated 226 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:45.560152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.571405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e01466-bff0-4c3f-a333-f6ad0d85366f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.610237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.610877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.611160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.611436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] PATH_INFO: `/v3/groups/40190e63afe24695970c19539106579f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.612632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] RBAC: Authorizing `identity:get_group(group_id=40190e63afe24695970c19539106579f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.612765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:45.614242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.616150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 234/453] 10.222.0.26 () {56 vars in 1180 bytes} [Sun Aug 7 00:55:45 2022] GET /identity/v3/groups/40190e63afe24695970c19539106579f => generated 229 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:45.622089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:45.659575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:45.672261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa81d305-92dd-4e7c-a723-1c9647a51d9d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.709162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.709687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.709962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.710230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.710955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] RBAC: Adding query filter params (group.id=fb2509f84bdd4fe68f8d8c125cbdb891, role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=95caef1c516c49b598639b368c2ec7ad) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:45.711257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.712746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b36d54-8a8a-4830-b0f7-f5e5b8554de2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.718711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 220/454] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:45 2022] GET /identity/v3/role_assignments?group.id=fb2509f84bdd4fe68f8d8c125cbdb891&scope.project.id=95caef1c516c49b598639b368c2ec7ad&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 611 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:45.737970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067b76e-e111-4239-937e-76216365c7cc None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:45.780376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:45.781008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:45.781291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:45.781568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:45.782308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] RBAC: Adding query filter params (group.id=40190e63afe24695970c19539106579f, role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=ae3747bdb31f45b9bc97b22fb3bfe021) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:45.782618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:45.784087 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443876fc-998a-489f-a0e8-fcf38d9972dc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:45.804278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 235/455] 10.222.0.26 () {56 vars in 1376 bytes} [Sun Aug 7 00:55:45 2022] GET /identity/v3/role_assignments?group.id=40190e63afe24695970c19539106579f&scope.project.id=ae3747bdb31f45b9bc97b22fb3bfe021&role.id=b0ec0093717c472da3640ec7121208d4 => generated 611 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:48.547139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e483aa-b1b9-4f0c-a170-383ba1be2ae5 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.547859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e483aa-b1b9-4f0c-a170-383ba1be2ae5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.547859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e483aa-b1b9-4f0c-a170-383ba1be2ae5 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:48.550350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 221/456] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:48.556179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65153c5f-c0b2-4cdd-9592-f37e74c3c263 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.557430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65153c5f-c0b2-4cdd-9592-f37e74c3c263 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.557430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65153c5f-c0b2-4cdd-9592-f37e74c3c263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:48.589104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-65153c5f-c0b2-4cdd-9592-f37e74c3c263 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:48.590831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-65153c5f-c0b2-4cdd-9592-f37e74c3c263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:48.682399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 236/457] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:48.731297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e483aa-b1b9-4f0c-a170-383ba1be2ae5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:48.853779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35a76a-bd56-4c8a-a1e8-187056ee0621 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.853996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35a76a-bd56-4c8a-a1e8-187056ee0621 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.854501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35a76a-bd56-4c8a-a1e8-187056ee0621 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:48.857428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 237/458] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:48.861971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7af287-e6bb-4916-b44f-d8148f595e86 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.862441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7af287-e6bb-4916-b44f-d8148f595e86 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.862858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7af287-e6bb-4916-b44f-d8148f595e86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:48.871877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e483aa-b1b9-4f0c-a170-383ba1be2ae5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:48.896564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-aa7af287-e6bb-4916-b44f-d8148f595e86 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:48.899161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-aa7af287-e6bb-4916-b44f-d8148f595e86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:48.903171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:48.930475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:48.932059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:48.932796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.933238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.933647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:48.947268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:48.949004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8a09a949-cfe2-4aac-863e-1430e9f896ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:48.964285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:48.965253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:48.966103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:48.966103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:48.966103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:48.968366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 222/459] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:48.969830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de28ab2-8561-47f2-92a6-1bf389b989a5 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.970310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de28ab2-8561-47f2-92a6-1bf389b989a5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.970759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de28ab2-8561-47f2-92a6-1bf389b989a5 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:48.976181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 223/460] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:48 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:48.977286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de28ab2-8561-47f2-92a6-1bf389b989a5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:48.987411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 238/461] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:48.988983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:48.989339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:48.989684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.011226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:49.012947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:49.099729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de28ab2-8561-47f2-92a6-1bf389b989a5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.108847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 239/462] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:49.116263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.161298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.161976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.162501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.162786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.163444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:49.163750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.165693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.181910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 224/463] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:49.185521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.223663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:49.242434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:49.263856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0330a-47f9-4eed-b80d-5c25682ffd0a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.301474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.302522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.302522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.302725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.314380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.315827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.317021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac2e909-bdad-47b3-81ad-c17383f4708c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:49.333333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:49.334306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:49.335158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:49.335158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:49.335158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:49.339590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 240/464] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:49.340920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.360962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.362042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.362514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.363028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.375478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.377336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:49.394626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:49.395550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:49.396401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:49.396401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:49.396401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:49.403234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 225/465] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:49.403234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.452127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5e848a-868e-4909-a9bc-8771a4a94529 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.489169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbd4c13-5950-4e2e-9a55-bfc7318db7c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.489783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd4c13-5950-4e2e-9a55-bfc7318db7c4 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.490058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd4c13-5950-4e2e-9a55-bfc7318db7c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.490328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd4c13-5950-4e2e-9a55-bfc7318db7c4 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.490968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbd4c13-5950-4e2e-9a55-bfc7318db7c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.492439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbd4c13-5950-4e2e-9a55-bfc7318db7c4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.498941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:49.505331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:49.529148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 241/466] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:49 2022] POST /identity/v3/users => generated 246 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:49.531915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16f3798-33b9-4125-a2ac-508f74167b5d None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.532767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16f3798-33b9-4125-a2ac-508f74167b5d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.533310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16f3798-33b9-4125-a2ac-508f74167b5d None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.535972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 242/467] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:49 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:49.537435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16f3798-33b9-4125-a2ac-508f74167b5d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.549186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b35b5db-bc46-4f9d-804b-f4f7244b0fff None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.616863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.618426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.618946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.619883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.621620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:49.622121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.624987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2d97b0-2430-4e94-a6f3-eedcea2ed303 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.651817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 226/468] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:49.654361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.654813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.655812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.690804 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:49.693140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:49.727655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16f3798-33b9-4125-a2ac-508f74167b5d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.775823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.776510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.776788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.777056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.777730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:49.778089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.779517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.794235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 243/469] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:49.795464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.799464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 227/470] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:49.803937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:49.876381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:49.889637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:49.926674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42986bb-64c9-4266-bc60-eee887962a68 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.939337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab86c6f-2af0-40fc-957c-2b5703d3a2c9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:49.963919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.964568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.964846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.965116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.965848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.967359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.984362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61e6647-6139-4243-889e-5aab218a8bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:49.985342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61e6647-6139-4243-889e-5aab218a8bb6 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:49.985869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61e6647-6139-4243-889e-5aab218a8bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:49.986358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61e6647-6139-4243-889e-5aab218a8bb6 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:49.987318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61e6647-6139-4243-889e-5aab218a8bb6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:49.989146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61e6647-6139-4243-889e-5aab218a8bb6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:49.998955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 228/471] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:49 2022] POST /identity/v3/users => generated 248 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:49.998955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:50.032494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 244/472] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:49 2022] POST /identity/v3/users => generated 248 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:50.101928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453383ad-c15d-4551-82a6-c734ee4e9862 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.102603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453383ad-c15d-4551-82a6-c734ee4e9862 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.103076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453383ad-c15d-4551-82a6-c734ee4e9862 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.105096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 245/473] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:50.113390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.114051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.114547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.155101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7256a9eb-599d-41de-bbda-256576fe8af9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:50.157313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:50.158642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:50.199200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:50.200969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.202584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.202868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.216866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:50.218398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:50.229229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:50.239564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:50.240591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:50.241439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:50.241439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:50.241439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:50.245798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 229/474] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:50.256312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:50.316757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 246/475] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:50.342251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:50.402327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:50.412375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b556c1f-27d6-4b0b-87c1-e0f75fc43228 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:50.456997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:50.457626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.457912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.458193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.458856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:50.459166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:50.461305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:50.478868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 230/476] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:50.484972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:50.501284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380a602-71d1-40cb-9869-838a7a9d8132 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:50.544664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:50.585956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:50.586639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.586914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.587182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.588457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:50.589911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:50.606454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.sql.core [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 07 00:55:50.606454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 07 00:55:50.606454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [parameters: {'id': 'a86a01a70ce141a185ce68824dc047a1', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 07 00:55:50.606454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80179) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application result = self._query(query) Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application conn.query(q) Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application result.read() Aug 07 00:55:50.625056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return ref.to_dict() Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 07 00:55:50.626113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application next(self.gen) Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application yield resource Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application next(self.gen) Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self.session.rollback() Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self.force_reraise() Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise self.value Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application session.commit() Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 07 00:55:50.627238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._prepare_impl() Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self.session.flush() Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._flush(objects) Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application compat.raise_( Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application flush_context.execute() Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rec.execute(self) Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 07 00:55:50.628344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 07 00:55:50.629215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application result = self._query(query) Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application conn.query(q) Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application result.read() Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 07 00:55:50.630044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [parameters: {'id': 'a86a01a70ce141a185ce68824dc047a1', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:50.630815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 07 00:55:50.631615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:50.632466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 247/477] 10.222.0.26 () {60 vars in 1131 bytes} [Sun Aug 7 00:55:50 2022] POST /identity/v3/roles => generated 144 bytes in 285 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:55:50.632532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678ea679-0837-4db5-888c-731f7f5a302f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:50.643487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:50.694024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:50.694693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.694980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.695311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.696043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:50.698317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:50.734437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 231/478] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:50 2022] POST /identity/v3/users => generated 249 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:50.811689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a040a6f-e402-4b8f-bc94-cb12b56afe06 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:50.815657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:50.855584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de115e-5362-417d-b130-d365c6f2efbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:50.856253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de115e-5362-417d-b130-d365c6f2efbe None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:50.856545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de115e-5362-417d-b130-d365c6f2efbe None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:50.856826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de115e-5362-417d-b130-d365c6f2efbe None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:50.870071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31de115e-5362-417d-b130-d365c6f2efbe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-31de115e-5362-417d-b130-d365c6f2efbe None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:50.875062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:50.877394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:50.879426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 248/479] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:55:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:50.882969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:51.008302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:51.022606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f778c3-7317-4b73-885c-aa583a501262 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:51.060919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:51.061540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:51.061822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:51.062097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:51.062768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:51.063073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:51.064476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359b4c1c-869e-4fd5-94d8-ba1f4815da30 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:51.073856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 232/480] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:52.143110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c9ddf-5b8d-4456-ae1d-cfdf8b1bf846 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.143672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c9ddf-5b8d-4456-ae1d-cfdf8b1bf846 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.144292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537c9ddf-5b8d-4456-ae1d-cfdf8b1bf846 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.147644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 249/481] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:52.154100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.154560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.155038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.191376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:52.193234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:52.330167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 233/482] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:52.373296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c9ddf-5b8d-4456-ae1d-cfdf8b1bf846 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:52.506194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537c9ddf-5b8d-4456-ae1d-cfdf8b1bf846 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:52.560000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7481a15-8c5a-437e-b2c7-ea6ce6c3172e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:52.560708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7481a15-8c5a-437e-b2c7-ea6ce6c3172e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.560987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7481a15-8c5a-437e-b2c7-ea6ce6c3172e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.561261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7481a15-8c5a-437e-b2c7-ea6ce6c3172e None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.578485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7481a15-8c5a-437e-b2c7-ea6ce6c3172e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a7481a15-8c5a-437e-b2c7-ea6ce6c3172e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:52.583562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:52.584544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:52.585401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 250/483] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:52.592858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:52.759280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:52.776546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd31863-91b7-4ce0-967d-69e72e059dd0 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.776872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd31863-91b7-4ce0-967d-69e72e059dd0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.777107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd31863-91b7-4ce0-967d-69e72e059dd0 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.778813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 251/484] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:52.783024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b136b-8ccc-43ba-b443-c536c51ed8dd None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:52.788637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17391f9b-a455-4450-9208-e744e824ee22 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.788922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17391f9b-a455-4450-9208-e744e824ee22 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.789201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17391f9b-a455-4450-9208-e744e824ee22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.827586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-17391f9b-a455-4450-9208-e744e824ee22 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:52.830904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-17391f9b-a455-4450-9208-e744e824ee22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:52.850069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:52.850977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.851406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.851822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.852757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:52.853268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:52.854812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:52.872161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 234/485] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:55:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:52.876757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:52.880791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:52.982344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 252/486] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:52.984571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06158f84-8e9a-4ff4-8918-fde8fb750248 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:52.984880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06158f84-8e9a-4ff4-8918-fde8fb750248 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:52.985160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06158f84-8e9a-4ff4-8918-fde8fb750248 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:52.986840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 253/487] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:53.002632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.002885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.003227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.030400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:53.031282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:53.049600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fc855-48d2-446d-a5a6-5a09980aef09 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.091761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.092728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.093189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.093635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.111148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.112661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.120852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:53.135028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:53.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:53.137149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:53.137149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:53.137149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:53.137149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 235/489] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:53.137149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.137818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 254/489] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:55:53.138994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.256482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:53.266744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc06d6b-d349-4b60-9e74-b3b51fd0dacf None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.284256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cc9ce8-0cdb-4383-91c2-28d92666d5e1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.326302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.326956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.327238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.327515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.342475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.343085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.343364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.343637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.344339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:53.344622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.346037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.349147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:53.356662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:53.357581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:53.358363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:53.364282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 255/490] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:53.364282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.368608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 236/491] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:53.370377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.445716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:53.493391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f840f-5fb6-4973-919b-1c67df391955 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.497229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6979239-3a6f-448c-8069-49439263d559 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.534396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.534934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.535218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.535495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.536168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:53.536521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.537916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.541177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.542059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.542577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.543066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.553575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 237/492] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:53.553575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.560923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:53.566455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:53.567419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:53.568256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:53.572570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 256/493] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:53.573621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.578579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:53.687131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9735af3-398d-4964-9a75-48a6a1b3d1dd None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.693772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bad74a9-28e0-4fd6-9896-ecd6c5730cfb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.717720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:53.734169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.735072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.735511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.735704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.736158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.736363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.736467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.736726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.747178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.749197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.750352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.752327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:55:53.764363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:53.765545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 238/494] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:53.766461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-ed568184-674b-413c-9fda-47e1ddccdfdc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:53.772067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:53.773120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:53.773956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:53.773956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:53.773956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:53.773956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 257/495] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:53.775024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bab25-7f94-4b1d-86cc-9da6749b8593 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.775328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bab25-7f94-4b1d-86cc-9da6749b8593 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.775604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bab25-7f94-4b1d-86cc-9da6749b8593 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.776933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 258/496] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:53.777698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901bab25-7f94-4b1d-86cc-9da6749b8593 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.840255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:53.884025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d4724-8028-4ff7-b3c0-d2a74a838ee9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.889910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901bab25-7f94-4b1d-86cc-9da6749b8593 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:53.921899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.922520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.922803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.923077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.923767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:53.924071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.925563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7203d0a2-144e-4c60-98d0-f86643832005 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.927391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:53.928037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.928347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.928623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.929389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:53.929954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:53.931468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63488499-6ab7-4c19-a8c1-9712cffc3857 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:53.935001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 239/497] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9998f06-85f2-492c-ad8d-122a729c1440 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.936702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9998f06-85f2-492c-ad8d-122a729c1440 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.936996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9998f06-85f2-492c-ad8d-122a729c1440 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.938616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 240/498] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:53.941242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9998f06-85f2-492c-ad8d-122a729c1440 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:53.975118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 259/499] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:53.976706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:53.976991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:53.977270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:53.994993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:53.996217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:53.996900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.058997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9998f06-85f2-492c-ad8d-122a729c1440 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.070596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 260/500] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:54.071473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.100663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.101193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.101474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.101747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.102404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:54.102711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:54.104113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6f216d-89a2-4348-b328-0f3d0fc83f25 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:54.119938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 241/501] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:54.119938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.119938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.119938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.136363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.144045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:54.145373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:54.201594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a613ecf-d72c-4993-8887-d4e78f9c6383 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.240998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 242/502] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:54.241197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.241197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.241197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.241197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.241537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.252503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:54.254115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:54.270634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:54.272614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:54.273959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:54.273959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:54.273959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:54.273959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 261/503] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:54.273959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.286504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.374862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea257a50-981d-4cdc-a100-75b9ba90ee17 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.387666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.394684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291bd246-2b9b-4659-8ffc-3c8392fdf9e6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.416151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.416834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.417311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.417419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.430415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.431046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.431834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.432110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.446487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:54.452333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:54.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 262/504] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:54.454940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.504284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:54.506271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:54.538613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.573545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848ad3e-6f27-4b44-977a-a7d34c26ae61 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.615961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.616645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.616920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.617192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] PATH_INFO: `/v3/users/glance` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.620012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 243/505] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:54.621229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.627481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:54.628926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:55:54.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:54.646695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 263/506] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/users/glance => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:54.647609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.672990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.754970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c82781-8f35-43a4-b484-65a1c43a5acf None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.777705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609adb0f-17d6-4122-b182-a53fef9347ad None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.789402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.795781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.796476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.796752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.797017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.810016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:54.811491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:54.820922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:54.821534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:54.822301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:54.822301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:54.822747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:54.823726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:54.825030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:54.826700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:54.827634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:54.828706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:54.828706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:54.828706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:54.828706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 244/507] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:54.828706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.844565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 264/508] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:54.845522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:54.957612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:54.964095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede8bdfe-90ac-40c4-83d4-ad347cff4dab None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:54.990908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebb0835-e462-49e8-ae0c-8ae9d3e3b35a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.009050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.010307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.011204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.011204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.011524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:55.011820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.013559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.026341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 245/509] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:55.026341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.046414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.047044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.047397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.047671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.061981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.063476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:55.079648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:55.080732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:55.081574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:55.081574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:55.081574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:55.081574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 265/510] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:55.081792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.093420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.162471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eedec92-0678-4b65-aee2-042404e2810a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.204208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.204881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.205161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.205436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.206172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:55.206485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.207979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.209692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c01d07-e828-446a-ae87-3d7a8f3f3b09 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.240785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.249271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 246/511] 10.222.0.26 () {56 vars in 1137 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:55.249494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.249976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.250136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.250259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.250596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.251273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:55.251556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.252969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.266933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 266/512] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:55.267795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.385753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe531bc-fe19-4ee8-b217-51ce861e28bb None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.388375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.408979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d80ff-6234-42a5-953e-ffacdc025445 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.430627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.431798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.432068 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.432373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.439957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.441494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.448363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.449006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.449287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:55:55.454300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:55.455315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 247/513] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/users/cinder => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:55.456299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.465178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.466603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:55.486732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:55.487898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:55.489021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:55.489021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:55.489021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:55.489379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 267/514] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:55.490091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.539348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.584521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3e55db-00db-489d-bf3c-f65915c23638 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.611040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f9520e-18fb-49fe-b5f0-7453789f1535 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.623897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.624543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.624828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.625104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.625745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:55.626070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.627455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.639816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 248/515] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:55.640717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.657756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.658370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.658663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.658916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.673222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.746686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.748109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.763035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82f6282-0edc-42d5-926c-74a2f7d6fb25 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:55.796055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.804453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:55.805330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:55.805609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:55.805874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:55.807143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:55.808682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:55.834001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 249/516] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:55:55 2022] POST /identity/v3/users => generated 247 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:55:55.835806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.879199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 268/517] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:55.880196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:55.965810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:55.996382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996acedf-ac7d-4608-b11e-72ceefa4becb None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.040376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2352e08-e6d2-4ee1-b33e-09f55f3eb729 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.086261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.087056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.087434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.088329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.089248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:56.089677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.091845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.099347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.099965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.100363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.100647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.101305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:56.101607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.103016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.116291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:56.126900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 269/518] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:56.128766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.129851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 250/519] 10.222.0.26 () {56 vars in 1137 bytes} [Sun Aug 7 00:55:55 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:56.130744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.281643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d4558b-df21-49fa-b734-2e56dd5b04cd None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.285624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e50a4-7407-4cb7-bc17-cabcf82633ac None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.331406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.331971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.332302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.332588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.334336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.335041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.335325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.335954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.341290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.342782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.343990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:56.350511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.351983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:55:56.358133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:56.359280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:56.360333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 270/520] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:56.360864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:56.379922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:56.381266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:56.382717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:56.382717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:56.382717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:56.384842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 251/521] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:56.385761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.447665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:56.508263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53da1f7-1590-424b-be1b-5236dc4541fa None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.537625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250e761d-36d2-4274-afed-6c4ec99f7e8d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.556078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.556971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.557353 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.557726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.566261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.570934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:55:56.584798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:56.586021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:56.587021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 271/522] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/users/neutron => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:56.590313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.592508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.592872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.592997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.594087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.673263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:56.691570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.693228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.732428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814a64df-5e26-4f64-acb1-90dc7c73c70a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.756106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:56.777478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:56.778298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:56.778584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:56.778861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:56.779570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:56.779875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:56.781457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:56.807305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 272/523] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:56.807305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.837565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 252/524] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:56.837565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:56.959422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:56.969173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aae62f-eda2-4222-a9b8-b48627db419f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:56.974204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcf41fa-6b2f-47db-b570-5f75cd151923 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.024502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.025550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.025977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.026396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.027302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:57.027566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.027678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.028383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.028895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.029494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.029595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.030846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:57.031495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.034214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.049740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 253/525] 10.222.0.26 () {56 vars in 1139 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:57.050586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 273/526] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:57.051465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.059723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.105172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:57.205132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca9ead1-7b6a-47fe-8711-25bdc3c2cbbb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.218550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b48e13-fddf-435b-aec8-25c92dbe64a3 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.253253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.253883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.254163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.254433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.265430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.266865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.270110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.270883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.271311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.271751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.272640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=9f6e6a410517438885907f00fabe80f0) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:57.273506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.275567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:57.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:57.282082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:57.283139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:57.283139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:57.283139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:57.286639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 274/527] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:57.287355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.296538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 254/528] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/role_assignments?user.id=9f6e6a410517438885907f00fabe80f0&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:57.297901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.336648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:57.424597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:57.441050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2d7504-56ce-47a3-bf2b-a874cd9485f4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.464838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfb2475-20c7-41d3-930b-0e75f742cc06 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.488399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.489652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.490079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.490689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.500621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.502854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.513208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.513981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.514364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.514734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:55:57.531361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:57.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:57.534937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:57.538633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 275/529] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:57.541612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.649591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.653287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.671794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:57.713580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b66925-eec0-4887-9943-fffe119d3441 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.755636 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.756550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.757010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.757359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.759029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:57.759029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.760185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.774563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 276/530] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:57.780179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.798012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:57.805790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 255/531] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:55:57.806965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:57.907355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be8325-9b58-46df-a5f0-d6a5dc2e5790 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.967306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:57.968594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:57.969089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:57.969567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:57.971152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:57.971984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:57.974471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:57.984105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23771ba-4b18-4648-afcd-2c26682b5bbb None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:57.992426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 277/532] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:57.994382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.001267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:58.039253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.041535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.042080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.042568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.058032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.060486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:55:58.074798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:58.075946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:58.077651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:55:58.084196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 256/533] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:58.084196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.098466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:58.135046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72120d-c9b1-4349-b5ca-6f46bd896ed7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.173884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.174674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.175056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.175453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.187190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.188916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.198280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc0a4e-872c-4978-bf38-e65f78aa609f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-eaf32785-68dd-42a9-8ab1-c80d8e1acd8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:58.206841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:58.208886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 278/534] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be138e8-64aa-4f6a-b282-6bb7dde7884b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be138e8-64aa-4f6a-b282-6bb7dde7884b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.211622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be138e8-64aa-4f6a-b282-6bb7dde7884b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.213399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 279/535] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:58.214199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be138e8-64aa-4f6a-b282-6bb7dde7884b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.240368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.240881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.241160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.241433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.242167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=7bd8b2ff3ce14290ac0cf476021e3496) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:58.242475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.244486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.260271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 257/536] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/role_assignments?user.id=7bd8b2ff3ce14290ac0cf476021e3496&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:58.260271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.301651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:58.394912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c095cc3-34bc-4914-b419-a8d892e5e7e7 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.411023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be138e8-64aa-4f6a-b282-6bb7dde7884b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.444682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.445619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.445905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.446417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.447757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:58.447970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:58.448354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.450177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de9a33a-2ec4-4498-95c8-2fb01841126d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.460654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.461266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.462013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.462013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:58.463363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.464865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.469020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 258/537] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:58.470842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.471244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.471677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.481600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 280/538] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:58.482540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.498956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:58.500132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:58.588381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 259/539] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:58.589693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.611749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2d6b3c-a5e1-4007-b69a-1350234beb2b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.636506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:58.652732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.653364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.653639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.653909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.661374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.662953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:55:58.675866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:58.676810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:58.677640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 281/540] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:58.678094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.734540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:58.750645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8539b1-f031-4b96-9331-c7217a80f136 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.822540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.824151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.824773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.825455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.827937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=f2a3c404c5f840368281af8532b7b46b) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:58.828628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:58.832668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:58.833526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7accde-8b96-4e7e-9790-c01663ad6e3e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:58.851137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 260/541] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/role_assignments?user.id=f2a3c404c5f840368281af8532b7b46b&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:58.851137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:58.914927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:58.915609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:58.915926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:58.916201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:58.946032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:55:58.959299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:58.960480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:58.961562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 282/542] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:58.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:59.038659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:59.056282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b403db69-4094-49f3-88b2-1089760e21e1 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:59.109848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:59.110725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.111155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.111576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.112607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:59.113065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:59.114846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:59.137404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c357a32b-4e24-49db-92be-cb85c61cb77d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:59.145775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:59.160949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 261/543] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:59.168206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:59.192661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:59.193371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.194035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.194314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.195032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:59.198253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:59.199870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:59.222370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 283/544] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:55:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:59.229018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:59.387314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab023d89-4699-4324-bd5a-51c3ed789ab5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:59.434717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3588bc5e-6223-4aef-9251-6cd54659f1e3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:59.453422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:59.454455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.454954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.455435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.456881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=8a22fb90c01a4c689550c0efc1fbaf71) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:59.457505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:59.460296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:59.471245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:59.471641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 262/545] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:55:59 2022] GET /identity/v3/role_assignments?user.id=8a22fb90c01a4c689550c0efc1fbaf71&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:55:59.472121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.472606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.486029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:59.487697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-dd212ed7-7ea9-44e7-9598-a95ee233a4b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:55:59.504619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:55:59.506419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:55:59.508226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:55:59.508226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:55:59.508226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:55:59.511216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 284/546] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:55:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:55:59.517893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:59.551414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:59.625801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c5757d-67af-426e-97c1-5b4786c62d33 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.626851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c5757d-67af-426e-97c1-5b4786c62d33 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.627456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c5757d-67af-426e-97c1-5b4786c62d33 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.629484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 285/547] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:55:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:55:59.636734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.637397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.637881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.679283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:55:59.681924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:55:59.695392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4941946e-7ee9-4f3e-930e-a35004e11da2 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:59.722379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:59.747936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:59.747936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.747936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.748387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.749062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:55:59.749372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:59.752086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:55:59.782738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 263/548] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:55:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:55:59.788736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:59.846947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 286/549] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:55:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:55:59.875647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:55:59.889591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:55:59.909288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedc719e-0b3b-40d3-8be6-9605f45aeb50 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:55:59.962399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:55:59.964008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:55:59.964521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:55:59.964985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:55:59.980109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:55:59.983438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:00.017135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:00.020001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:00.021774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:00.027480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 264/550] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:55:59 2022] GET /identity/v3/users/swift => generated 83 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:00.035825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:00.143286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38684fa4-fd0f-4e80-8dd3-4ab768d475c7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:00.168660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee2b66e-6698-4201-b944-dd32240b45a5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:00.209997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:00.210626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.210900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.211172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:00.211914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:00.212222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:00.214207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:00.216138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:00.237939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 265/551] 10.222.0.26 () {56 vars in 1135 bytes} [Sun Aug 7 00:56:00 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:00.238600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:00.239321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.239601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.239871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:00.243567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:00.271889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:00.279397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:00.280557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:00.281361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 287/552] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:55:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 389 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:00.291658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:00.310429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:00.470381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2707e76-8a57-4838-a1c5-89d346701040 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:00.479200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05979d45-ba0d-443f-8514-a1d8352c01e4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:00.516271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:00.517358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.517835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.518330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:00.519250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:00.519842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:00.521900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:00.531600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 288/553] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:00.537442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:00.543588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:00.544215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.546778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.546778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:00.616609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:00.624916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:00.626537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:00.722750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:00.753550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 266/554] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:00.761311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:00.786273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d069f9-8e3d-42f7-b1e2-4bd892d563b9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:00.842852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:00.843664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.844068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.844526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:00.873612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:00.876132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-623c39e7-80aa-42d6-93c7-cbdbf3bac95b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:00.916822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:00.918640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:00.920369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:00.920369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:00.920369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:00.923867 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 289/555] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:00.926422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216ae502-50db-4655-9fab-8a01197d5619 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.927224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216ae502-50db-4655-9fab-8a01197d5619 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.928113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216ae502-50db-4655-9fab-8a01197d5619 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:00.931365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 290/556] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:00.932947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216ae502-50db-4655-9fab-8a01197d5619 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:00.952090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d24eeb9-cc80-4251-bc74-2e4a10759413 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:00.967761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:00.990820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:00.991501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:00.991882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:00.992411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.003746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.006048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-b9b9584a-450b-4ca1-bcfb-e9054d8187f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:01.026165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:01.028410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:01.031225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:01.031225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:01.031225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:01.031225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 267/557] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:01.032168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.032603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.032977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.052257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:01.053643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:01.141159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216ae502-50db-4655-9fab-8a01197d5619 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.160118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 268/558] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:01.161900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.186317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:01.207581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.209265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.210081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.210873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.212433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:01.213306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.216454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.239512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 291/559] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:01.241752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.289297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:01.322709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abe7ad-1ff0-4a1d-a751-a55c7ce58c54 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.365260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.365918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.366206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.366469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.367162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:01.367470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.369769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.386105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 269/560] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:01.387002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.433611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a88976-87f2-40a4-a29d-96a1611bc965 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.491224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.492882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.493628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.495037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.508362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53325d61-5747-4594-908c-8d4a7b43884d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.513313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:01.523731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:01.525949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:01.527574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:01.533482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 292/561] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:01.535280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.550912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.551658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.552006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.552317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.555238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:01.572478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.575735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:01.598789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:01.600752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:01.602590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:01.604167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:01.616485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 270/562] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:01.618529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.672745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe437b-73f7-4925-8ce8-b9546ed0f19f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.710023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.710731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.710921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.711261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.718616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.720075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:01.732562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:01.734879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 293/563] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:01.736835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.796719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622d1b16-91f6-449b-9f33-37292d3834f6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.840739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.841367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.841644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.841916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.842614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:01.842919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.844642 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab36faa0-33d7-4fba-93de-30b17ea97741 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.867779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1395d843-0229-4da6-94fc-0ec84e023a06 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:01.868750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 271/564] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:01.870349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f56ca6e-8464-4bdc-88b9-683f199d36a1 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.870643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f56ca6e-8464-4bdc-88b9-683f199d36a1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.870922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f56ca6e-8464-4bdc-88b9-683f199d36a1 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.871256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:01.873514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 272/565] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:01.873759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f56ca6e-8464-4bdc-88b9-683f199d36a1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.924654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:01.925439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:01.925785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:01.926139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:01.926919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:01.927220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:01.928776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:01.951969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 294/566] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:01.959043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:01.960891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.018512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f56ca6e-8464-4bdc-88b9-683f199d36a1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.057665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.058192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.058469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.058743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.059442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:02.059747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:02.061282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19d6978-3e16-4caa-8faa-3c63e6bb9a1c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:02.075234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 273/567] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:02.077408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb86a9-8bda-4faa-9688-30fe3d1d9443 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.077893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb86a9-8bda-4faa-9688-30fe3d1d9443 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.078373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bb86a9-8bda-4faa-9688-30fe3d1d9443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.092460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24515eea-85e8-42db-a5ca-47a01eb294d6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.102023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-35bb86a9-8bda-4faa-9688-30fe3d1d9443 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:02.103927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-35bb86a9-8bda-4faa-9688-30fe3d1d9443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:02.134426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.134840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.135111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.135378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.155868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:02.157441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:02.181481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:02.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:02.193299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:02.193299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:02.193299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:02.193299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 295/568] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:02.193299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.193818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 274/569] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:02.193818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4030e44e-7bb5-4b40-855b-d8ebdef79511 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.193818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4030e44e-7bb5-4b40-855b-d8ebdef79511 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.193818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4030e44e-7bb5-4b40-855b-d8ebdef79511 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.193818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 275/570] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:02.193818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4030e44e-7bb5-4b40-855b-d8ebdef79511 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.225448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.265902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.334499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4030e44e-7bb5-4b40-855b-d8ebdef79511 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.348003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe42cf-0f09-4a26-b7a5-c101bca71961 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.374556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.375178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.375590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.375923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.376919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=7342064620f34dabb93dd10f92f444a4) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:02.377742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:02.379230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:02.385786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 276/571] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/role_assignments?user.id=7342064620f34dabb93dd10f92f444a4&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:02.386678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.399389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817ab83-d926-4998-959b-99878963f335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.401525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817ab83-d926-4998-959b-99878963f335 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817ab83-d926-4998-959b-99878963f335 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.402210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817ab83-d926-4998-959b-99878963f335 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.417124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817ab83-d926-4998-959b-99878963f335 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:02.418762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817ab83-d926-4998-959b-99878963f335 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-0817ab83-d926-4998-959b-99878963f335 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:02.437160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:02.438407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:02.439615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:02.439615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:02.439615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:02.445871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 296/572] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:02.445871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.445871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.445871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.468280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:02.470458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:02.540261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4bf966-1d01-4d45-a7b3-4c76e1005b78 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.549686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.554746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.568859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 297/573] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:02.570070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.609657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.610248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.610682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.610971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.612442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:02.612756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:02.614489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:02.634646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 277/574] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:02.635765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.730794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e504fe9a-200f-43a2-a12c-f2bb503edd90 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.775991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.777076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.777351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.777724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.793012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f61c90-4911-4bbb-824a-bee5b8305d22 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:02.793221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:02.798944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:02.799953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:02.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 298/575] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:02.806112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.842158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:02.842799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:02.843072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:02.843349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:02.844007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:02.844344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:02.845784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:02.861843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 278/576] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:02.862488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:02.895254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.910166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:02.968178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc30228c-cc40-48e6-a9bd-626974dd56c4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.026698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.027613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.028058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.028536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.041620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacf599f-1514-49ff-8ef4-0e4b9ad76d65 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.047868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:03.057611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:03.059188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 299/577] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:03.062208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.098861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.099513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.099794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.100079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] PATH_INFO: `/v3/users/glance` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.112620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.114660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:03.135554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:03.136859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 07 00:56:03.138457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:03.144559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 279/578] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:02 2022] GET /identity/v3/users/glance => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:03.146016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.233415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7130eb-a3bb-454c-b5bf-a179c5f6427d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.258993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:03.264587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:03.276407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d09fe-2a66-4c0c-95a7-3adbb13e87dd None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.294132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.294747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.295093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.295334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.295961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:03.296286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.297810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.308701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 300/579] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:03.309448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.326792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.327481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.327767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.328039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.336278 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.338076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:03.349944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:03.350910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:03.351901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 280/580] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:03.352433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.425280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa8a580-e159-4a12-873a-3b48e649c1b6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.463578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e52866-4f7e-4443-bca3-ee469e5327cc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.466542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.467234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.467519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.467799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.468576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:03.468885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.470410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.479985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 301/581] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:03.480948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.508854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.509505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.509790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.510100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.510799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:03.511106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.512560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.526159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:03.537674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 281/582] 10.222.0.26 () {56 vars in 1137 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:03.538373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.550821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:03.604345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c197165-f1b3-4d87-aafc-fb595ffecd3d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.656700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c202be-38f0-433c-83ef-2cdceb94c7b9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.670665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.671720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.672199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.672904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.691146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.693753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.693957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.694568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.694895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.695166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.695946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:03.696219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.698238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.716603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 282/583] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:03.717896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:03.719734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:03.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:03.721581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:03.721581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:03.721581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:03.721760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 302/584] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:03.725196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.800190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:03.825571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:03.836895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2af3166-7bcd-48bf-bc29-5f928ca4a95f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.841554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ca706-b676-4c33-bcf2-94589fde2465 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:03.876545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.877599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.878097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.878583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.879771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:03.880453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:03.880736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:03.881009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:03.892709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.893681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:03.894810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.895148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:03.910863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:03.911873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:03.912806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:03.912806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:03.912806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:03.912806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 283/585] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:03.912806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:03.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:03.915092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:03.916599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:03.916599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:03.916599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:03.916599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 303/586] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:03.916821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.047996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95b0114-bf56-41b4-8838-926f7ab0df04 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.055666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b3b2d9-7f54-4b01-a1f5-ff4ece427593 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.091977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:04.110976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.111925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.112423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.112884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.113590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.113925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:04.114415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.114617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.115054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.115481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/9f6e6a410517438885907f00fabe80f0/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.118040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.139091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=9f6e6a410517438885907f00fabe80f0, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.145148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 304/587] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:04.145328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.145439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.148206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:04.168525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 284/588] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:03 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/9f6e6a410517438885907f00fabe80f0/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:04.169516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.305154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14abd18a-fa42-4ac5-88d6-7b9517fac6cb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.333857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe0506e-a16c-41ce-a14b-aff905e3830c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.352598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.354405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.354978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.355485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.356811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:04.357350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.359334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.377371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 305/589] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:04.378742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.405586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.406720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.407190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.407721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.408969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:04.409509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.412461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.429741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 285/590] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:04.430993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.470701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:04.472626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:04.530162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a8f28-6961-479f-94f2-d3a9fd677ae4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.564269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c593f4-8c25-4a30-8da1-ce6a6fe663fd None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.570869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.571560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.572485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.572808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.583160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.584912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:04.598166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:04.599366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:56:04.600612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:04.604413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 306/591] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/users/cinder => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:04.605376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.606000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.606892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.607177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.607455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.616424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.617886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7cfd1c81-421d-4a69-ba1c-ab9192ebc865 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:04.629578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:04.630582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 07 00:56:04.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:04.636121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 286/592] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/users/neutron => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:04.637562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0a2af-18e7-4e9e-bc20-0424bbbf81e0 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.637857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0a2af-18e7-4e9e-bc20-0424bbbf81e0 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.638172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce0a2af-18e7-4e9e-bc20-0424bbbf81e0 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.639469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 287/593] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:04.640178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0a2af-18e7-4e9e-bc20-0424bbbf81e0 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.739245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79d4b38-ca35-43f8-bf7a-9085b52b9868 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.753445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:04.765939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce0a2af-18e7-4e9e-bc20-0424bbbf81e0 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.770379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:04.784662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.785356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.785641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.785916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.794213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.795681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.803938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.804680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.809266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.809266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.809266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:04.809266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:04.809620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:04.810583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:04.812147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:04.812599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc67bdd-d6cb-4f3f-98b3-82d5428c4fb0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:04.814630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 307/594] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:04.815354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.831675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 288/595] 10.222.0.26 () {56 vars in 1137 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:04.837396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.838084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.838084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.859261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:04.860543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:04.935000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0858f68-7eff-484c-ae94-0982748f37a7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:04.948306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 289/596] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:04.948306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:04.985045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:04.985662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:04.985943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:04.986211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:04.986952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:04.987256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:04.988724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.012854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 308/597] 10.222.0.26 () {56 vars in 1139 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:05.012854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.061165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.067598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.084992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a6ead0-53b2-4aa9-8b8b-39209390e66b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.126185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.126967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.127351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.127738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.128471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:05.128850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.130287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.132693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9187d7-158c-4275-84c5-e4cbfd161f11 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.143382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 290/598] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:05.144269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.175577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.176210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.176514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.176792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.188281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.189738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:05.205319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:05.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:05.207295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:05.207295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:05.207295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:05.207295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 309/599] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:05.207295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.261382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60096a4-a5a9-4322-8a2b-72d525bba496 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.300210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.300943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.301324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.301605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.312060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.317235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:05.323060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:05.324305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:05.325246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:05.326023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.327866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 291/600] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:05.329033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.348029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b603fc-2ac4-43cd-b55d-28fe570d4254 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.388033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.388796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.389152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.389433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.400862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.402365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:05.418081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:05.419297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:05.420183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:05.420183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:05.420183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:05.420183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 310/601] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:05.420461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.453408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12233e2-5b92-44a4-90f8-227b7475542b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.492273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.492919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.493201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.493668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/7bd8b2ff3ce14290ac0cf476021e3496/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.510585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=7bd8b2ff3ce14290ac0cf476021e3496, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.512042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.540271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 292/602] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:05 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/7bd8b2ff3ce14290ac0cf476021e3496/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:05.540271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.551762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01afed88-8900-4dd1-bb22-074c1e8efc5c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.593953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.594471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.594756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.595034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.595873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:05.596195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.597656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.606846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.614531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 311/603] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:05.615777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.626776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.686874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4c2c3c-d765-49ad-8fa2-9022d5055d32 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.742488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.743118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.743399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.744193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.744916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:05.745223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.746644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.755697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 293/604] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:05.756611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.768198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37dde1d-e9de-456b-bbea-8787a3255d0f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.808089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.808765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.809047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.809315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.810084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:05.810393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.811897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.825468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 312/605] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:05.826529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.891364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d6c601-1ddd-4628-b2d1-580b79e6ebc8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.904692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.905297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:05.935379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.936066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.936397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.936730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:05.944321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:05.945857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:05.951879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ab2bf3-e758-44be-b865-aebdcd808ba4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:05.961071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:05.962322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:05.963286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 294/606] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:05.964309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:05.998009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:05.999193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:05.999476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:05.999753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.015546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.017121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:06.038367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:06.039490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:06.040559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:06.040559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:06.040559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:06.044393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 313/607] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:06.045462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.108886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5972264-0aaf-4e76-a57c-896490bfb830 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.167902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.169218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.169890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.170559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.172215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86423b3b-676a-4d34-a24e-d42b8c493c08 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.183402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.186481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.191395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:06.205024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:06.206185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:06.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:06.209033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 295/608] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:06.209977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.225301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.225924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.226359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.226764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.227479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:06.227785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.230095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.246853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 314/609] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:06.248069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.254493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:06.333190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbaf9ac-1bb8-4dde-9044-77f08340fa17 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.363585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7550822b-d52a-4494-bb3b-fda9a4f13a7c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.371184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.371808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.372089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.372398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.373048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:06.373356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.374864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.393463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 296/610] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:06.394420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.404544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.405576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.406118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.406678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.407565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:06.408161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.409872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.424044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 315/611] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:06.425253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.473573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:06.512969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:06.524249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479c154-3254-4709-8853-10e339ac8640 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.563088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224c8019-2e22-4511-9c50-eb1343b91bb8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.565135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.565859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.566262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.566428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/f2a3c404c5f840368281af8532b7b46b/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.585811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=f2a3c404c5f840368281af8532b7b46b, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.587321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.612356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 297/612] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:06 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/f2a3c404c5f840368281af8532b7b46b/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:06.613892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.618526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.619765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.620052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.620375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.632142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.633842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-93ade791-fc7d-475a-9d1a-8d67e3d771c6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:06.653954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:06.655029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:06.655981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 316/613] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:06.657150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b635c3f-d2a8-4b10-b035-912e3403d17d None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.657272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b635c3f-d2a8-4b10-b035-912e3403d17d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.657497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b635c3f-d2a8-4b10-b035-912e3403d17d None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.663535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 317/614] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:06 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:06.664605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b635c3f-d2a8-4b10-b035-912e3403d17d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:06.778371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70d23c1-e04d-4f12-9dd4-ce2168ada73f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.802690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:06.819198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b635c3f-d2a8-4b10-b035-912e3403d17d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:06.834519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45805951-70d6-48b3-933e-faad4dc1da5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.835822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45805951-70d6-48b3-933e-faad4dc1da5f None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.836113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45805951-70d6-48b3-933e-faad4dc1da5f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.836440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45805951-70d6-48b3-933e-faad4dc1da5f None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/8a22fb90c01a4c689550c0efc1fbaf71/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.852776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:06.865247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45805951-70d6-48b3-933e-faad4dc1da5f None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=8a22fb90c01a4c689550c0efc1fbaf71, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.866759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45805951-70d6-48b3-933e-faad4dc1da5f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.879383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:06.880701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.882306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.883495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.884904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:06.885335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:06.886955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:06.893690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 298/615] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:06 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/8a22fb90c01a4c689550c0efc1fbaf71/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:06.895917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:06.896533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:06.896960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:06.922873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:06.924168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 318/616] 10.222.0.26 () {56 vars in 1135 bytes} [Sun Aug 7 00:56:06 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:06.924818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:06.930186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.037179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 299/617] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:07.062221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.159861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ac59b3-ef51-4141-815a-ca6da7558735 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.201120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ce1ee-9425-4f7e-90fa-2a9350009c4d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.206004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.206811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.207194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.207583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.240067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:07.243300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:07.249861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:07.269936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:07.271133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:07.272196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:07.272196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:07.272196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:07.272196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 319/618] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:07.277662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.282069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.282733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.283096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.283379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.288641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:07.312502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:07.317454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:07.318552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:07.319372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:07.324761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 300/619] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:07.330831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.446457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d6826-7f7a-430c-82e3-8789275f6869 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.477867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a77f878-b2ad-4891-a3b5-1ab1fef32ea1 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.516859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.517481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.517765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.518044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.518789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:07.519098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:07.520550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:07.534720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 320/620] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:07.540347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.566851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.567728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.568183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.568657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.569471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:07.569916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:07.576273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:07.602527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 301/621] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:07.609218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.627111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:07.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5ac948-3bda-4276-b168-7de4793808e5 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.712127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.712994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.713283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.713557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.722234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:07.723763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:07.734113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:07.738650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:07.739551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:07.740470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 321/622] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:07.747747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.819301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36156fb5-8985-4024-8dca-2074afb11a58 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.861032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.862133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.862421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.862748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.878193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:07.880385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-23e65adb-53c8-4460-a8e1-bdd2a6c0f02e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:07.900679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:07.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:07.904841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:07.904841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:07.904841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:07.904841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 302/623] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:07.904841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87766284-258c-4c3c-99d4-a89a1cddd628 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.904841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87766284-258c-4c3c-99d4-a89a1cddd628 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.905563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87766284-258c-4c3c-99d4-a89a1cddd628 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.908300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 303/624] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:07.908300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87766284-258c-4c3c-99d4-a89a1cddd628 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:07.938543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21aff77-0df1-4c78-a81f-d183700efe2a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:07.975388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:07.995162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:07.995774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:07.996049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:07.996349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:07.997048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:07.997347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:07.998814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9650b7ee-8e00-44c0-8994-88b5d1cb5ff0 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.011722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 322/625] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:08.013055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.013344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.013620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.029936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.032923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:08.034149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:08.043093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87766284-258c-4c3c-99d4-a89a1cddd628 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.082701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.083616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.084101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.084575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.085489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:08.085959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:08.087753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.102266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 304/626] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:08.103379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.118040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 323/627] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:08.119616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.213266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41891a4-7c2e-42e9-ab24-d7a9fd3b8acf None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.232206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5421d125-8cdb-49f4-8913-996dd30b2205 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.251397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.253989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.254851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.255285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.255777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/7342064620f34dabb93dd10f92f444a4/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.272178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.272632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=7342064620f34dabb93dd10f92f444a4, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:08.273136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.273517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.273885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.277319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.286596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:08.288231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.295620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 305/628] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:08 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/7342064620f34dabb93dd10f92f444a4/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:08.296827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.307674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:08.314416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:08.315460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:08.316460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:08.324916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 324/629] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:08.330858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.420160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fef118c-b48e-4aaf-9643-3d31a0f69d87 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.459187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.459820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.468554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.468554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.472182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:08.477231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:08.478780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:08.479602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 306/630] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:08.489587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.507109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8c144e-ea24-4634-a28b-171f98415bf9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.546625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.547766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.548288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.548761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.549708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:08.550242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:08.552004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.574393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 325/631] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:08.580554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.621092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.633343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.691333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00daea8-b3b3-4af5-9905-6ef98b3e8eb9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.715799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9165aa29-5fa2-49f2-b97f-3695d8f2863d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.738937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.739882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.740373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.740825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.741683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:08.742214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:08.744380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.754144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 307/632] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:08.760350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:08.761436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.762229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.762565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.762923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.861839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:08.864523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:08.889137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68708c6-c2c7-4967-9526-db6d6920fdfb None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:08.901959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.920540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:08.967111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:08.967679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:08.967964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:08.968257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:08.984741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 326/633] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 07 00:56:08.990899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.001384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.002866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:09.041966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:09.043170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:09.044742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:09.044742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:09.044742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:09.044742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 308/634] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:09.053300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.105137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565cfc9f-f784-44b2-b877-e309e43edc2e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.143017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.143660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.143938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.144208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.146776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:09.156862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.158583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b1fe0797-8d01-4cf1-9a24-cb7ab554e78c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:09.177116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:09.179374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:09.181891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:09.181891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:09.181891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:09.181891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 327/635] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:09.185048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da4a6a-2a90-43dc-be27-ed32103619ca None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.185310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da4a6a-2a90-43dc-be27-ed32103619ca None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.185588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da4a6a-2a90-43dc-be27-ed32103619ca None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.187040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 328/636] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:09 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:09.187623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da4a6a-2a90-43dc-be27-ed32103619ca None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.268819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8386ed-e39a-4812-987f-dc03f1fa351f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.315043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da4a6a-2a90-43dc-be27-ed32103619ca None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.329603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.330367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.330644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.331327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.332312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:09.332623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.334497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55068ef-e5ae-4dc1-9ec2-d04e58731905 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.352128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 309/637] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:09.352128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.352128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.352741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.358317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.359348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.360364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.360682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.361347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:09.361650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.363057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c60526-18b4-4854-8f94-91d5764d12f4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.372644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:09.373916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:09.376316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 329/638] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:09.377576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ee219-4182-4a95-9578-197fe70b9615 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.377860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ee219-4182-4a95-9578-197fe70b9615 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.378189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ee219-4182-4a95-9578-197fe70b9615 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.379553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 330/639] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:09.380098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ee219-4182-4a95-9578-197fe70b9615 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.423192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:09.442795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:09.482622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 310/640] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:09.484501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.519986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ee219-4182-4a95-9578-197fe70b9615 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.579664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.580503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.580851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.581191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.589634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.591219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-48556607-9b21-47f1-a911-8f2f82bad070 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:09.605172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:09.606807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 331/641] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/users/glance => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.612499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.613218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.633779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:09.635625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:09.647860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83aaee-c747-4796-97fa-3aee8d3aa1d9 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.706141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.707035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.707487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.707894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.717691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.719662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.722351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 332/642] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:09.723217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:09.745929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:09.747618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:09.748665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:09.751724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 311/643] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:09.753416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.780392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:09.801036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:09.874549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ddcde2-1c96-4bba-92c8-b04af3669c66 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.914914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7a04ea-5f40-404c-9022-fccc59446031 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:09.927039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.927682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.927962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.928705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.949438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:09.954378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:09.955526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 333/644] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:09.956672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:09.966730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:09.967719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:09.968222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:09.968600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:09.970407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:09.970407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:09.971271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:09.996997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 312/645] 10.222.0.26 () {56 vars in 1137 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:09.998035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.132309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:10.141352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:10.142104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb736005-7e2a-4e28-a0c5-bad60f4d7873 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:10.166437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383a5b3-5d7b-4f37-a2f4-8da57eb3077e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:10.199246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:10.200276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:10.200721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:10.201119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:10.217785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:10.218043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:10.218410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:10.218696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:10.218960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:10.219666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:10.219977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:10.221513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:10.223465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:10.225501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 334/646] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:10.230395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.249798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 313/647] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:10.249798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.376114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b636fac2-0002-4a8d-b152-4dcdb96126ba None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:10.415019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf502b40-b259-47d7-99c2-336a954fb970 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:10.435962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:10.437010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:10.437500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:10.437957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:10.439360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:10.439900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:10.442443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:10.457925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 335/648] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:10.460868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.473533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:10.474533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:10.475001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:10.475475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:10.501037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:10.507736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:10.605825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:10.608840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:10.650731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91daa528-6fa8-4f25-a81a-9a289a9070e1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:10.698905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:10.699601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:10.699794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:10.700068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:10.700708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:10.701019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:10.702512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:10.711964 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 336/649] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:10.712897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.784165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 314/650] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 539 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:10.785638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.856485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:10.876592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:10.884790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f85cf9f-176b-415e-91a2-a4168938801f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:10.949232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:10.949893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:10.950179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:10.950454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:10.951235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=9f6e6a410517438885907f00fabe80f0) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:10.951544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:10.953045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:10.959577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 337/651] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:10 2022] GET /identity/v3/role_assignments?user.id=9f6e6a410517438885907f00fabe80f0&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:10.961011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:10.962245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0abb6da-eeaa-434b-ae82-594dedb6f8dd None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.020847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:11.021927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.022406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.022861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.044693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:11.047599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:11.079176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:11.081318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:11.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:11.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:11.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:11.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 315/652] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:11.083703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:11.141649 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5546fb15-1f1f-43a6-bdf9-2d378bc247b9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.206660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:11.207711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.208198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.208778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.229710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:11.232147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:11.236619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-3a179e9a-0e48-4d3f-bec7-b1b0aa132a37 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:11.252823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:11.254206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:11.255197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:11.255197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:11.255197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:11.255197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 338/653] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:11.255435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff26687-4b74-430c-9224-e0ffad045dc4 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.256655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff26687-4b74-430c-9224-e0ffad045dc4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.256655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff26687-4b74-430c-9224-e0ffad045dc4 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.259900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 339/654] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:11.259900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff26687-4b74-430c-9224-e0ffad045dc4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:11.282976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:11.292624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc905b2-3497-4d10-8937-590a8e252a04 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.363764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:11.364858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.365339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.365793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.386322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:11.388972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:11.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:11.416557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:11.418890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:11.418890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:11.418890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:11.423990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 316/655] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:11.425693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:11.468520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff26687-4b74-430c-9224-e0ffad045dc4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.536196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:11.537275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.537763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.538233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.539364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:11.539887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:11.542993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f744a54-5c68-41e5-9eae-f9009608c5fa None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:11.572285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 340/656] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:11.572285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.572285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.572285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.607419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:11.608999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:11.628044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccefc56-2f48-4da9-9de5-4da19a68ef76 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.700206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:11.715276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:11.716338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.716932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.717287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.718456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:11.719103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:11.721553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:11.727010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 341/657] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:11.736852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:11.747039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 317/658] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:11.748644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:11.772318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:11.921472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c79beb-6c7f-463a-8136-26f312e89987 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.950078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada547c4-0a2d-402d-a862-673cf220e5f3 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:11.985644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:11.986245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:11.986528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:11.986812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:11.997436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:11.999423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:12.018246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:12.020059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 318/659] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:11 2022] GET /identity/v3/users/cinder => generated 84 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:12.021923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.030548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:12.031239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:12.031521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:12.031796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:12.032494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:12.032842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:12.034285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:12.057424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 342/660] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:12.058901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.164934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:12.213196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9536c4-c143-4deb-8d31-3494a04741ed None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:12.256211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:12.257088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:12.257553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:12.257922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:12.269271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:12.271293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:12.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:12.289897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 319/661] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/users/neutron => generated 85 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:12.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.335285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:12.439152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a033f5-95a3-4a96-9c65-424c22b96079 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:12.481229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621513e4-1dd3-496f-8785-b57208aec12f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:12.504949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:12.506034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:12.506535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:12.506960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:12.527260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:12.536487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:12.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:12.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:12.539098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 343/662] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 481 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:12.541382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:12.541864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.562433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:12.563458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:12.563934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:12.564445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:12.565832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:12.566383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:12.569106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:12.602788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 320/663] 10.222.0.26 () {56 vars in 1137 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:12.604259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.708403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202984-b0b3-4368-ab37-72f1b5a16aa4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:12.719811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:12.780583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:12.781737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:12.782251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:12.782712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:12.789184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6188a80-a0db-4130-99ea-37de7917155d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:12.796098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:12.798911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:12.821170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:12.822859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:12.823858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:12.824313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 344/664] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:12.825934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.850318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:12.851382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:12.851892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:12.852537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:12.855779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:12.855779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:12.858472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:12.903150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 321/665] 10.222.0.26 () {56 vars in 1139 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:12.903150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:12.990805 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:13.031792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b6176-e79c-4d27-ab61-9e79fae28a3a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:13.089088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d70f460-02be-413b-b9fc-9db436c36f32 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:13.100843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:13.101759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:13.102102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:13.102389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:13.103254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:13.103577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:13.105058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:13.116716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 345/666] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:13.123499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:13.164420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:13.165338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:13.165850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:13.166296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:13.175621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:13.331325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:13.333386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:13.343625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef85b75-e30e-4316-8ba2-1c7dbfe347a3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:13.395383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:13.396490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:13.396853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:13.397321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:13.398561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:13.399084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:13.402088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:13.416658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:13.436773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 346/667] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:13.436773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:13.533604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 322/668] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 636 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:13.535821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:13.607653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:13.631217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae1689-4792-45da-af6f-0b06f83108eb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:13.716432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:13.717471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:13.717940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:13.718389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:13.745466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1195c30-bc43-489e-9c78-ddf07defaf94 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:13.816070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:13.816070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:13.816070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:13.816070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:13.834240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:13.837060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:13.847519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:13.860083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:13.861884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:13.864186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:13.864186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:13.864186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:13.866872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 323/669] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:13.868051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:13.873023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:13.875868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.040348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:14.049967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7696151e-879c-499d-8793-89f4a64267ce None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:14.066201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 347/670] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 635 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:14.068976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:14.122586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:14.123603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.124071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.124579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.126000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=7bd8b2ff3ce14290ac0cf476021e3496) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:14.126551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:14.129319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.138040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 324/671] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:13 2022] GET /identity/v3/role_assignments?user.id=7bd8b2ff3ce14290ac0cf476021e3496&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:14.139632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:14.207273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305c1b30-cc67-4c2a-9974-04f2438bb752 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:14.253812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:14.266105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:14.267048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.267528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.267982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.288873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:14.292670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:14.320545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:14.322913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:14.328049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:14.328049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:14.328049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:14.328049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 348/672] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:14.328049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:14.354237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f9af91-49bd-4990-8319-93783d630b4c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:14.408705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:14.423789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:14.425310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.426068 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.426551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.427892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:14.430109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:14.433174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c921e-f702-4dd8-9bb6-46759b1f86d7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.457761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 325/673] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:14.460033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe8f6cd-19ff-4eda-b741-4ef05c450d24 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.460565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe8f6cd-19ff-4eda-b741-4ef05c450d24 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.461219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe8f6cd-19ff-4eda-b741-4ef05c450d24 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.463942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 326/674] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:14.465499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe8f6cd-19ff-4eda-b741-4ef05c450d24 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:14.525305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f369a-181d-42d7-87c0-96154f53b86c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:14.588520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:14.590105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.590847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.591513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.612935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:14.616769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.642838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe8f6cd-19ff-4eda-b741-4ef05c450d24 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:14.650694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:14.652751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:14.654431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:14.654431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:14.654431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:14.660540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 349/675] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:14.663470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:14.682854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:14.699577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:14.700225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.702247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.702247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.702247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:14.702247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:14.703372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c20b78-0d5d-40ee-9acf-5c02597c2ba6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.728317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 327/676] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:14.728317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.728317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.728317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.756407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:14.758297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:14.861037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:14.880028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760340bb-f88d-4a03-8bb5-a1d47e75b68d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:14.900462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 328/677] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:14.900462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:14.957683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:14.958446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:14.958902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:14.959386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:14.975978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:14.978710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:14.997616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:14.999023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:15.000113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 350/678] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:15.001142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:15.111756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50f903-d2ec-464a-9729-45db5ee59bff None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:15.136527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:15.182631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:15.183478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:15.184443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:15.184748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:15.185442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:15.185788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:15.188172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:15.213277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 329/679] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:15.214891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:15.222120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af7b59-85ef-4d9d-afa8-8dc2e7d08eb7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:15.296891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:15.298880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:15.306382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:15.306382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:15.313684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:15.315573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:15.349132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:15.350877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:15.352682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 351/680] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:15.353781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:15.377557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:15.421250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff4752e-5718-4748-8bda-dc9fbed29bb5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:15.478937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:15.479716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:15.480060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:15.480398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:15.494483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:15.500403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:15.502445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 330/681] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:15.504036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:15.536557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63212459-e4f1-4434-949f-28072b9f467b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:15.550656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:15.603032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:15.604214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:15.604759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:15.605235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:15.606552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:15.607058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:15.609173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:15.641637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 352/682] 10.222.0.26 () {56 vars in 1135 bytes} [Sun Aug 7 00:56:15 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:15.643033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:15.691531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0835e08a-4c19-405a-92c6-4f6db520e15c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:15.765843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:15.767602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:15.768320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:15.769138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:15.783300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:15.786843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:15.795599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:15.814087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:15.816173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 331/683] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:15.818201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:15.869144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0d6b01-a804-48fa-8783-9b6daecc85cb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:15.925909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:15.927083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:15.927718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:15.928195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:15.929716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:15.930508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:15.933258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:15.953229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 353/684] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:15.957541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:16.000924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:16.017926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ad62a-4872-4e70-96da-06d5f98eaa2c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:16.084113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:16.085193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:16.085693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:16.086189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:16.087408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:16.088017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:16.090607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:16.115133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 332/685] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:16.115133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:16.152148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77658595-bf41-4df1-84a7-409ddc1ddf6d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:16.202154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:16.212570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:16.213278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:16.213721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:16.214071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:16.345640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d0a1bc-c839-4840-8485-a27ecae88a02 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:16.346275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:16.348204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:16.403823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:16.405091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:16.405902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:16.406482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:16.407809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:16.408600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:16.411493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:16.434824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 333/686] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:16.436963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:16.460142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:16.521412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 354/687] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 560 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:16.521412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:16.567706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:16.689089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d977a7-dc13-41d5-8324-6418eb6fd028 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:16.747000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:16.747871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:16.748427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:16.748847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:16.761288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e453887d-050a-4115-a5bc-297ffe711123 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:16.763671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:16.765319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:16.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:16.782855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:16.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:16.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:16.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:16.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 355/688] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:16.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:16.813044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:16.813893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:16.814228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:16.814628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:16.815643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=f2a3c404c5f840368281af8532b7b46b) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:16.816115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:16.818063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:16.825211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 334/689] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:16 2022] GET /identity/v3/role_assignments?user.id=f2a3c404c5f840368281af8532b7b46b&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:16.826246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:16.927489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:16.953241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:16.965674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb23767-a86e-4993-8e4c-03dcd847f46c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:16.987367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d603041d-1173-4883-81ad-92b9bc8a4c7f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.032446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.033086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.033380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.033736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.034673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=8a22fb90c01a4c689550c0efc1fbaf71) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:17.035205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.036930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a891f97e-2503-4327-9357-3c0142dc30db None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.038583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.039783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.039783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.039783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.047108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 356/690] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:16 2022] GET /identity/v3/role_assignments?user.id=8a22fb90c01a4c689550c0efc1fbaf71&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:17.048511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b561d1-a86e-476b-aae6-ad9702fb53a7 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.048804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b561d1-a86e-476b-aae6-ad9702fb53a7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.049083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b561d1-a86e-476b-aae6-ad9702fb53a7 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.052578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 357/691] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:17.052578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b561d1-a86e-476b-aae6-ad9702fb53a7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.058481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.060067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-5854c2c2-99ea-469a-9715-d775b2e37314 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:17.085945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:17.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:17.089345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:17.089345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:17.089345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:17.089345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 335/692] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:17.091458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.091893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.092130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.118962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:17.122082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:17.233319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 336/693] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:17.234380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.242789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b561d1-a86e-476b-aae6-ad9702fb53a7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.309434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.310425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.310910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.311402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.312844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:17.313378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.315632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.330243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 358/694] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:17.331664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.346538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:17.353812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:17.392830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a17a5a-825a-4be8-858e-23e4be79fc80 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.442998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.443767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.444071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.444337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.444950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:17.445408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.447556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.465637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 337/695] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:17.466566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.536910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567a43db-30ff-4323-b422-0dd81174d1d7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.580737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.581515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.581913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.582316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.596479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.598237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:17.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:17.617461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:17.618391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:17.618391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:17.618391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:17.618391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 359/696] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:17.618896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.619323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520f0e9c-d6a3-4b43-ab22-1fdcf27ac0de None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.647294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:17.669284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.670373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.670849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.671375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] PATH_INFO: `/v3/users/nova` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.680851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.682792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.695206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:17.700750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:17.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:17.702996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:17.705384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 338/697] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:17.706539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.747596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0397159a-916b-42f5-bc5a-af110d615d9c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.787404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.787783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.788071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.788411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.797035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.798627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:17.812551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:17.813674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:17.814665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:17.819022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 360/698] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:17.819551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.837882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78492da-37a1-4e20-8e9e-9885ed252c7b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.905949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.906995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.907427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.907846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.908689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:17.909174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.910902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:17.928044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 339/699] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:17.929227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:17.947656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89731fb-df91-4b19-b066-b15435ecdc56 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:17.968836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:17.989164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:17.989917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:17.990242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:17.990664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:17.992457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:17.992797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:17.995394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:18.009032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:18.016534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 361/700] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:18.017513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:18.129229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036edc97-6f39-4868-9eb0-5e8f265365fc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:18.148666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e2572e-f59e-4923-a806-6e8ffd0699a7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:18.193184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:18.194297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:18.194492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:18.194731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:18.195031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:18.196103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:18.196337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:18.197669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:18.198333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:18.199485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:18.200029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:18.200856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:18.202959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:18.218248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 340/701] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:18.219376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:18.251136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 362/702] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:56:18 2022] POST /identity/v3/users => generated 254 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:18.252272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:18.322390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:18.424841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b89625-bc60-41fe-a280-77db7c49e9e2 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:18.425575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:18.430896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef435094-c795-40ad-9920-f856267b7ebe None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:18.471180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:18.472165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:18.472689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:18.473185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:18.474219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=7342064620f34dabb93dd10f92f444a4) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:18.474777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:18.476555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:18.488757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 363/703] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:18 2022] GET /identity/v3/role_assignments?user.id=7342064620f34dabb93dd10f92f444a4&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:18.489147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:18.489994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:18.490330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:18.490606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:18.645348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:18.648840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:18.836430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:18.843813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 341/704] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 625 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:18.848935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:18.978546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:18.988763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb883f8a-cc82-4319-b17c-14d38943d894 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:19.046432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:19.047501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:19.047985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:19.048505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:19.070446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:19.073722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:19.106700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:19.108558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:19.110515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:19.110515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:19.110515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:19.110515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 364/705] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:19.119575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:19.320914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe58f07-489b-4a67-a336-1d85b8c8140b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:19.390219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:19.391212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:19.391605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:19.392079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:19.393321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:19.393787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:19.396496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:19.419336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 342/706] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:19.428521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:19.675721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8024714e-3251-45b2-b3f3-95c109a71245 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:19.754495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:19.755369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:19.755780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:19.756135 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:19.770723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:19.773128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-44596751-829a-4730-a6b5-0ffffe0f35a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:19.795114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:19.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:19.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 365/707] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:19.806680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:19.855186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca628d25-bcfd-4a74-9283-0b8e51f2b34e None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:19.855947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca628d25-bcfd-4a74-9283-0b8e51f2b34e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:19.855947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca628d25-bcfd-4a74-9283-0b8e51f2b34e None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:19.858388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 366/708] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:19.862814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:19.863175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:19.863421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:19.899710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:19.901032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:19.903495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:19.935781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:20.013824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844bf952-5501-4fe9-bbca-4ec2f43c4d7e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:20.105930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:20.106711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:20.107265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:20.108484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:20.112227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:20.112227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:20.114063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:20.132522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 367/709] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:56:20.145150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 343/710] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:20.155075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:20.195698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:20.514341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e04be3-890f-47c4-a414-ccf7680f84e4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:20.606321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4c35ba-3664-4ae0-839e-bfb9a56e5e64 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:20.629159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:20.630932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:20.631933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:20.633286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:20.635478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=9f6e6a410517438885907f00fabe80f0) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:20.636560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:20.640055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a6742c-0599-4bd2-8dd2-96fc57472563 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:20.657252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 368/711] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:20 2022] GET /identity/v3/role_assignments?user.id=9f6e6a410517438885907f00fabe80f0&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 503 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:20.663270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:20.663715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecbf9d-8ff7-4bf4-9479-80cea383b351 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:20.664003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecbf9d-8ff7-4bf4-9479-80cea383b351 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:20.664952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecbf9d-8ff7-4bf4-9479-80cea383b351 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:20.678465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:20.679549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:20.679996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:20.680771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:20.682498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:20.684813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 369/712] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:20 2022] GET /identity => generated 270 bytes in 23 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:20.687317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:20.687899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:20.688410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:20.693908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:20.710410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:20.726297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:20.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:20.730948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 344/713] 10.222.0.26 () {56 vars in 1129 bytes} [Sun Aug 7 00:56:20 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 534 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:20.745232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:20.758174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:20.760398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:20.962042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 370/714] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:20.998946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:21.013715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec38d9d-a0dd-44d0-bea9-4a0080110de7 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:21.083976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:21.087255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:21.090079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:21.090870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:21.092378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:21.093521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:21.097078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:21.114356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 345/715] 10.222.0.26 () {56 vars in 1124 bytes} [Sun Aug 7 00:56:20 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 372 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:21.124116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:21.223058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866a5eb7-5b30-465b-b32f-115d48d5e78d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:21.243199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:21.295707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:21.305818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:21.307497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:21.307865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:21.308180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] PATH_INFO: `/v3/services/network` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:21.310024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:21.312006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-24902920-7e82-40c6-9de3-bba34f728d30 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:21.335471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:21.336639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:21.338090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:21.340407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 371/716] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:20 2022] GET /identity/v3/services/network => generated 88 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:21.343028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac452a5-ebce-4773-a137-1d9bcbf908f9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:21.343623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac452a5-ebce-4773-a137-1d9bcbf908f9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:21.344480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac452a5-ebce-4773-a137-1d9bcbf908f9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:21.349396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 372/717] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:21 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:21.350922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac452a5-ebce-4773-a137-1d9bcbf908f9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:21.375585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c506d2-e6db-464d-bba5-99b6e09c3cc5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:21.449886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:21.451434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:21.452745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:21.453374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:21.455604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:21.456669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:21.459412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db272463-94e4-4568-b995-b39c0d8fafb6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:21.476103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 346/718] 10.222.0.26 () {56 vars in 1124 bytes} [Sun Aug 7 00:56:21 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:21.479520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:21.480332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:21.482038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:21.514796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:21.517378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:21.619129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac452a5-ebce-4773-a137-1d9bcbf908f9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:21.699905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:21.701293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:21.702425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:21.703231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:21.705042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] RBAC: Adding query filter params (name=network) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:21.705851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:21.708729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:21.720780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 373/719] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:21 2022] GET /identity/v3/services?name=network => generated 124 bytes in 371 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:21.731306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:21.783794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 347/720] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:21.813001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:21.852034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:21.950880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:22.000894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58933ed3-b8ae-4608-883b-533abf8b79e5 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:22.094326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:22.095598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:22.096932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:22.097672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:22.098715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] RBAC: Adding query filter params (type=network) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:22.100756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:22.103111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:22.116435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 374/721] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:21 2022] GET /identity/v3/services?type=network => generated 124 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:22.140563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135ebca7-8b02-4bde-902f-a35908678c7a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:22.242632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:22.244172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:22.244583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:22.244948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:22.272753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:22.278121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:22.280065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:22.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:22.284444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 348/722] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 467 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:22.296646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:22.544401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d13bb9-5b61-4f38-8d13-f814c08e6840 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:22.600705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:22.623702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:22.624998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:22.625468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:22.625913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:22.627208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:22.627717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:22.630555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:22.645829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 375/723] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:22.654111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:22.827349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0fd226-0fa2-4d8b-ab8d-72ecd3465c50 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:22.829253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:22.878167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:22.878818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:22.879096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:22.879410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:22.896186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:22.897806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-41722226-d0d8-49c6-aeda-b2a6ff4562dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:22.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:22.921507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:22.923526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:22.923526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:22.923526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:22.923526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 349/724] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:22.927314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:23.108503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a3a722-544b-4e21-a8a6-bd637457b845 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:23.108748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a3a722-544b-4e21-a8a6-bd637457b845 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:23.109205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a3a722-544b-4e21-a8a6-bd637457b845 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:23.111043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 350/725] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:23.116806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:23.117196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:23.117598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:23.138565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:23.139968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:23.159397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f152b9a-66c4-49cd-b2e2-41f9245f1eb7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:23.242446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:23.243080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:23.243356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:23.243848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:23.244666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:23.245002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:23.246638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:23.282318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 376/726] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:23.293415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:23.349906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 351/727] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 07 00:56:23.351820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:23.390285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:23.474811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0546ea8a-0247-4ad9-a0d3-5ff21913267a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:23.541325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:23.553818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd37802-fc31-4349-8073-3f7bb961c129 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:23.593736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:23.594415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:23.594735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:23.595093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:23.607954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:23.613186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:23.614931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:23.616521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 352/728] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:23.625925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:23.626244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:23.626949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:23.627437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:23.627910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:23.642786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:23.645422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:23.668469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:23.670228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:23.672000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 377/729] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:56:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 378 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:23.679258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:23.804506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d63ee2-a0e2-4d2d-90fd-66405a9e7fbc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:23.844686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:23.845314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:23.845592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:23.845853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:23.851196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:23.851196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:23.851196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:23.860273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 353/730] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:23.870797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:23.881341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:23.970202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8f69b-13b4-4297-92f8-abbbd16c4e7e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:23.992872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:23.998408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935ae089-6a6d-426e-a093-3dbcdef4b54c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:24.042438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.043473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.043942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.044446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.046595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:24.047082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:24.049600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:24.085812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 378/731] 10.222.0.26 () {56 vars in 1149 bytes} [Sun Aug 7 00:56:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 407 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:24.095263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:24.157039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.157700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.157990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.158261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.199268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:24.200800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:24.264397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:24.266255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:24.267917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:24.267917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:24.267917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:24.268280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fc4bf1-cbc7-4649-a584-3d9372a5b482 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:24.272364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 354/732] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 402 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:24.293999 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:24.308396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fe0f0d-f8a5-406b-ac90-90287cf14e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.309004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe0f0d-f8a5-406b-ac90-90287cf14e36 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.309288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe0f0d-f8a5-406b-ac90-90287cf14e36 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.309564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe0f0d-f8a5-406b-ac90-90287cf14e36 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.358537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:24.378720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:24.408887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fe0f0d-f8a5-406b-ac90-90287cf14e36 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:24.410568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fe0f0d-f8a5-406b-ac90-90287cf14e36 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:24.429528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45832153-ec1e-4957-b4d0-3cb78884c129 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:24.469833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.470469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.470747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.471020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.471673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:24.471971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:24.473400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df83ff20-54e0-4496-ac5f-af0b50d58c14 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:24.486902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 355/733] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:24.488530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0297ff-d3bc-4f33-9271-7411861ba425 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.488837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0297ff-d3bc-4f33-9271-7411861ba425 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.489187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0297ff-d3bc-4f33-9271-7411861ba425 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.490761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 356/734] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:24.496125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0297ff-d3bc-4f33-9271-7411861ba425 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:24.546635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 379/735] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 07 00:56:24.548228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.548550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.548904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.569749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:24.571414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:24.614151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0297ff-d3bc-4f33-9271-7411861ba425 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:24.642239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:24.663430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.664083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.664360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:24.664484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.664694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.675228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:24.676840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:24.682571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 380/736] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:24.683933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:24.703125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:24.704140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:24.705124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:24.712884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 357/737] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:24.720639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:24.862051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed3614f-7215-459b-93d8-2a1feff14a77 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:24.869420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca55b45-885b-4ea2-9cbf-b769ec377004 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:24.932775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.933576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.933925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.934277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.937024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:24.938379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:24.938614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:24.939062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:24.952771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:24.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:24.960931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:24.962118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:24.963726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:24.964887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 358/738] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:24.964887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:24.970908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-2e02cb66-74df-434a-81dc-4d7111008223 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:24.996807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:24.998170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 381/739] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f99693-92d7-4463-b2a4-313705fee214 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f99693-92d7-4463-b2a4-313705fee214 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f99693-92d7-4463-b2a4-313705fee214 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 382/740] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:25.005130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f99693-92d7-4463-b2a4-313705fee214 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:25.036730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:25.104996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:25.143014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ad54b3-3de1-46a9-b4f5-41195fb246e4 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:25.178794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f99693-92d7-4463-b2a4-313705fee214 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:25.205758 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:25.206772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.207063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.207345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.208138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:25.208482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:25.209993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f4f4f4-4175-45d5-b52d-1d1bc37406a6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:25.226474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:25.227100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.227381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.227632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.229106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:25.229618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:25.231648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:25.252347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 383/741] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:25.252347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:25.256634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 359/742] 10.222.0.26 () {56 vars in 1135 bytes} [Sun Aug 7 00:56:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:25.258975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.259566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.260060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.294752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:25.296866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:25.428366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 360/743] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:25.428366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:25.455508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7f5e1f-6477-4a40-9af4-bb974ac91128 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:25.472459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:25.480666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:25.525734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:25.526809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.527318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.527839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.529101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:25.529763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:25.532721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:25.565245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 384/744] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:25.566856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:25.614272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fb945a-d896-4b17-8451-e2e043cd8d16 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:25.676395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:25.677590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.678099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.678650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.696767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:25.698427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:25.718165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f54167d-7b37-4b95-86cc-fba4f048a0bd None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:25.722423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:25.724682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:25.726914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:25.726914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:25.726914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:25.726914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 361/745] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:25.726914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:25.770590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8b0853-43fa-4e72-9867-dfcb4f117b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:25.771620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8b0853-43fa-4e72-9867-dfcb4f117b35 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.772097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8b0853-43fa-4e72-9867-dfcb4f117b35 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.772639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8b0853-43fa-4e72-9867-dfcb4f117b35 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.833516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:25.835595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:25.875118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd021bc-a91a-4b1e-9cde-1bf1944b8ae9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:25.895150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8b0853-43fa-4e72-9867-dfcb4f117b35 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:25.897074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8b0853-43fa-4e72-9867-dfcb4f117b35 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:25.933990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:25.934574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:25.934935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:25.935267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:25.935944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:25.937441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:25.955250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 362/746] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:56:25 2022] POST /identity/v3/services => generated 251 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:25.956100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.105431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 385/747] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 540 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 07 00:56:26.108289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5a454-cbec-4f12-b0c9-25929ee4a1da None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.108970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5a454-cbec-4f12-b0c9-25929ee4a1da None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.109553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5a454-cbec-4f12-b0c9-25929ee4a1da None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.112560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 386/748] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:26.113952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5a454-cbec-4f12-b0c9-25929ee4a1da None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.161260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175c0e28-c48e-4de2-b90f-a30aecc46a36 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.219233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:26.221532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:26.223036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.224023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.224349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.229264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:26.243210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:26.247693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:26.273666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:26.274900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:26.276106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:26.277946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 363/749] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:26.278331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.292629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5a454-cbec-4f12-b0c9-25929ee4a1da None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.356724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:26.357741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.358225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.358703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.359882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:26.360448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:26.363016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36840fed-d6f4-4cc5-a908-432624554725 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:26.385451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 387/750] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:26.394584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.395090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.395582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.429048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:26.431953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:26.456001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8270d623-9a51-4411-828a-afbfff6a1c33 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.517053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:26.517934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.518288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.518654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.536335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 388/751] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:26 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 07 00:56:26.536335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.543145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:26.545317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:26.570579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:26.572961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:26.576386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:26.576386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:26.576386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:26.576386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 364/752] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:26.576386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.636505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:26.657331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:26.715716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3b806f-01b8-47e8-a0b8-be1d1d869229 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.732097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead631cf-0106-43fa-b978-6f6da2c18d26 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.771539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:26.772069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.772447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.772860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.779631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:26.780489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:26.780921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:26.781325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:26.782161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:26.782595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:26.784525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:26.784843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:26.787400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:26.804281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 389/753] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:26.804281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:26.807638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:26.808632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:26.809568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:26.816513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 365/754] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:56:26 2022] GET /identity/v3/users/nova => generated 82 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:26.817790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:26.954337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86a46d3-0147-483e-a9a2-417b3b8df20b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.976308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:26.980788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6ccd87-9259-46c0-8170-df14ae520ce7 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:26.992672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:27.000389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.001021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.001305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.001579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.002253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.003786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.033861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 390/755] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:56:26 2022] POST /identity/v3/services => generated 240 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:27.033861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.044911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.045680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.046027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.046451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.047250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:27.047587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.049277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.064911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 366/756] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:27.065785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.182266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffaac3e-ead3-4f93-80ff-0ce97ad9000b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:27.226790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f166eb-615d-46f5-85ae-7a61c3737f3d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:27.249596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.250490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.250792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.251185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.252789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=4814b0fcdd08486bb0f7ae5d8486b9f9) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:27.252789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.255096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.263329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 391/757] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/role_assignments?user.id=4814b0fcdd08486bb0f7ae5d8486b9f9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:27.264473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.273721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.274538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.274978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.275376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.276304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:27.276844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.278548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.313512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 367/758] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:27.320811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.338197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:27.379717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:27.451150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a676d3-5131-432d-abc0-31a2e8825494 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:27.510099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.511499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.511895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.516287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.523181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.525903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.542286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c34b04-3c81-4549-b976-7a6512c730eb None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:27.547836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:27.549289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:27.550371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:27.556544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 392/759] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:27.563472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.673012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.673770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.674145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.674515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.699895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d4b6d-3e38-473a-8cb6-55a38a307c91 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:27.704788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:27.716675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.719817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.746679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.747496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.750099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.750498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.751345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:27.752671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.755318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:27.763496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:27.764716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:27.766295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:27.766295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:27.766295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:27.773857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 368/760] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 454 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:27.777064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 393/761] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:27.781160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.782823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:27.903574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc2031d-607e-4279-ae39-ce0e845f8dd4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:27.944327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:27.944892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:27.945176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:27.945454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:27.946376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=7342064620f34dabb93dd10f92f444a4) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:27.947299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:27.948943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:27.956059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 394/762] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/role_assignments?user.id=7342064620f34dabb93dd10f92f444a4&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:28.031078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:28.081113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c9800-11e1-434a-a7e1-517b63ba0945 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:28.138300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cffbe87-9004-483e-baec-134152989115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:28.139062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cffbe87-9004-483e-baec-134152989115 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:28.139372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cffbe87-9004-483e-baec-134152989115 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:28.139650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cffbe87-9004-483e-baec-134152989115 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:28.140539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cffbe87-9004-483e-baec-134152989115 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:28.140911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cffbe87-9004-483e-baec-134152989115 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:28.142383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cffbe87-9004-483e-baec-134152989115 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:28.157827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 369/763] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:28.166564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:28.206393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:28.287947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7fab15-872e-4272-8dc1-5e29ce877e0b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:28.326229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:28.326871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:28.327152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:28.327429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:28.335027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:28.336520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:28.348902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:28.349852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:28.351166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 395/764] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:28.357883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cffbe87-9004-483e-baec-134152989115 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:28.511182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cffbe87-9004-483e-baec-134152989115 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:28.555302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:28.564078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:28.564747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:28.565013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:28.565288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:28.565990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:28.566293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:28.567765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c808b33-46f2-4991-abe8-d4c9bc78114c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:28.584811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 370/765] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:28.591060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:28.633479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d44ca0e-cac5-4bd5-9c17-2fed6e0f4f28 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:28.635458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d44ca0e-cac5-4bd5-9c17-2fed6e0f4f28 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:28.635940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d44ca0e-cac5-4bd5-9c17-2fed6e0f4f28 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:28.637945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 371/766] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:28 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:28.649879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:28.650450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:28.650898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:28.677744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:28.683141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:28.788822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:28.814648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f31bbd-5b9c-412b-a595-2f8ecfd4fe43 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:28.819781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 372/767] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:28.859113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:28.862652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:28.864953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:28.865239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:28.865509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/9f6e6a410517438885907f00fabe80f0/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:28.867674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=9f6e6a410517438885907f00fabe80f0, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:28.869876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:28.892223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 396/768] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:28 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/9f6e6a410517438885907f00fabe80f0/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:29.006474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7c4132-bf91-4025-9de0-1b7cfc164e0d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:29.043840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:29.058778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:29.059486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:29.059767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:29.060036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:29.060658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:29.062065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:29.071979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:29.073927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:29.075633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 373/769] 10.222.0.26 () {56 vars in 1129 bytes} [Sun Aug 7 00:56:28 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:29.079882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:29.330127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677ce9f2-2966-4287-8544-2ba6ed9dfab0 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:29.383684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:29.384430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:29.384719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:29.385002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:29.385689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:29.385995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:29.387394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:29.393864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 397/770] 10.222.0.26 () {56 vars in 1124 bytes} [Sun Aug 7 00:56:29 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:29.400728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:29.626713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235f0ad4-1550-4de5-9d89-0b4c49f87ca3 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:29.679395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:29.680015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:29.680319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:29.680978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:29.681699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:29.682002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:29.683477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d49cf72-d88d-4b07-a95b-a5eb0544798c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:29.693078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 374/771] 10.222.0.26 () {56 vars in 1124 bytes} [Sun Aug 7 00:56:29 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:29.699950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:29.743182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:29.818471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac09c35-2440-44f5-96d2-be237b725b09 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:29.819580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac09c35-2440-44f5-96d2-be237b725b09 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:29.819580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac09c35-2440-44f5-96d2-be237b725b09 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:29.820677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 375/772] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:29.829891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:29.830179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:29.830454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:29.885829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:29.886945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:29.924063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05f06e4-9b76-4e1f-9140-7f9589e5343b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:29.989267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:30.001507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:30.002129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.002414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.002687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.003400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] RBAC: Adding query filter params (interface=public, service_id=061d7b66b635488fbce91c2417f0e747, region_id=RegionOne) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:30.003707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:30.005119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:30.023356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 398/773] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:56:29 2022] GET /identity/v3/endpoints?service_id=061d7b66b635488fbce91c2417f0e747&interface=public®ion_id=RegionOne => generated 194 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:30.032278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:30.033327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 376/774] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:30.112569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:30.161283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:30.214791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42afc9-9fee-47b7-bd59-14e50f65e738 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:30.285227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:30.285727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.286000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.286272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.286881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:30.288278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:30.301814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 399/775] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:56:30 2022] GET /identity/v3/services => generated 772 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:30.318123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b969ed4-2b7b-439a-92eb-1efa6cf05e63 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:30.370829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:30.371493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.371772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.372041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] PATH_INFO: `/v3/services/network` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.372751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:30.374170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-c75c6800-9d9c-46b7-b269-14d4e3c77a17 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:30.388108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:30.389982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:30.391625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 377/776] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:30 2022] GET /identity/v3/services/network => generated 88 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:30.396129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:30.626598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1f9ba0-562b-4935-9e0f-f535e3966210 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:30.658907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8ec949-3a63-4b7c-8202-0ef9e52e1f5c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.660101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8ec949-3a63-4b7c-8202-0ef9e52e1f5c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.660755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8ec949-3a63-4b7c-8202-0ef9e52e1f5c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.663286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 378/777] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:30.669865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.670626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.671103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.688486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:30.689074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.689362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.689637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.690279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] RBAC: Adding query filter params (name=network) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:30.690583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:30.691981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:30.694283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:30.695833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:30.698100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 400/778] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:30 2022] GET /identity/v3/services?name=network => generated 124 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:30.703877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:30.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:30.800267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:30.823903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 379/779] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:30.858125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a010d55-af0f-43c0-a0a1-8e66c633be99 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:30.865172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:30.909203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:30.910256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:30.910667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:30.910959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:30.911613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] RBAC: Adding query filter params (type=network) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:30.911920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:30.913563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:30.921455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 401/780] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:30 2022] GET /identity/v3/services?type=network => generated 350 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:30.927780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.005220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:31.032387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de7be4d-b6b9-453f-af89-d0beefe82701 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:31.049496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911fd157-d836-46ff-96b8-6278985384cb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:31.092009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:31.093226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.093715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.094190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.095138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] RBAC: Adding query filter params (interface=public, service_id=c7e233ad9b8e4d92acf68b6a833ab219, region_id=RegionOne) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:31.095696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:31.097421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:31.106330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 402/781] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:56:30 2022] GET /identity/v3/endpoints?service_id=c7e233ad9b8e4d92acf68b6a833ab219&interface=public®ion_id=RegionOne => generated 194 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:31.113818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.137480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:31.140503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.141107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.141628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.170492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:31.182181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:31.183182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:31.184007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 380/782] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:31.202926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.273452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a5250c-8458-4631-b017-098d48d0bc82 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:31.304396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:31.314532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:31.323162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b29f2c6-5a5a-4a3b-a72b-6f1a1e434dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:31.323814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b29f2c6-5a5a-4a3b-a72b-6f1a1e434dc9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.324137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b29f2c6-5a5a-4a3b-a72b-6f1a1e434dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.324444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b29f2c6-5a5a-4a3b-a72b-6f1a1e434dc9 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.325166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b29f2c6-5a5a-4a3b-a72b-6f1a1e434dc9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:31.326595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b29f2c6-5a5a-4a3b-a72b-6f1a1e434dc9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:31.333495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 403/783] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:56:31 2022] GET /identity/v3/services => generated 772 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:31.340750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d696ab79-4b4b-4dd6-ab61-96316ac23d4c None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.341105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d696ab79-4b4b-4dd6-ab61-96316ac23d4c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.341381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d696ab79-4b4b-4dd6-ab61-96316ac23d4c None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.343216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 404/784] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:31 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:31.349658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.349969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.350273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.387569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:31.388849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:31.390757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b56b7c-1926-4ef9-9d64-17ba3b594ccd None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:31.459085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:31.460120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.460824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.461113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.463121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:31.463469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:31.466216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:31.485239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 381/785] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:31.492898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.522144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 405/786] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:31.568393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.703156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:31.714816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16983197-59fc-4508-9e87-a965d265b299 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:31.780797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a9a51d-1c28-418d-87d1-2efb12c0e960 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:31.798784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:31.800736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.801548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.802273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.820612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:31.821078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:31.822599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:31.833313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab0993d-d31f-4a47-a588-f0dc13bf6f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:31.834140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab0993d-d31f-4a47-a588-f0dc13bf6f78 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.834517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab0993d-d31f-4a47-a588-f0dc13bf6f78 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.834954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab0993d-d31f-4a47-a588-f0dc13bf6f78 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-a9e6284c-2f3f-4dea-b7b8-9855eac3cab2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:31.840584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:31.842141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:31.843407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:31.843407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:31.843407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:31.843407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 382/787] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:31.843407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84553235-5268-402d-9c32-0106ea118437 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.843726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84553235-5268-402d-9c32-0106ea118437 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.843801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84553235-5268-402d-9c32-0106ea118437 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.845444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 383/788] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:31.848205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84553235-5268-402d-9c32-0106ea118437 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.857304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab0993d-d31f-4a47-a588-f0dc13bf6f78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-8ab0993d-d31f-4a47-a588-f0dc13bf6f78 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:31.862856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:31.863872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:31.864774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 406/789] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:31.866363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:31.866718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:31.867062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:31.889517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:31.890989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:31.971156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 407/790] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:31.972143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:31.999959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84553235-5268-402d-9c32-0106ea118437 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.049992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.050630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.050912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.051377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.052105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:32.052443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.054053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.068424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 384/791] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:32.069451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.082881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:32.099889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea6bd5c-b848-446b-bcdf-8f2c639348de None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.117752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:32.143438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.144349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.144738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.145112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.145917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:32.146391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.148063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.159670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 408/792] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:32.160747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.224818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e48dfc-8a5f-4139-827a-359f39fc868c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.283768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.284840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.285465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.285910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.293043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266e4bc7-450f-4cef-9689-84f05b50bb69 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.312629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:32.323334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:32.325035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:32.327089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:32.332850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 385/793] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:32.334489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.348976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.348976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.348976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.348976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.362053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.364122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:32.382322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:32.385115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:32.387319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:32.388610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 409/794] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:32.396536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.440045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:32.472779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:32.537948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4caf0a-c05f-45e6-a71e-23423e48f02d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.555206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39fbb06-59b6-47d8-b5cb-8c6bd17bae99 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.588525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.588525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.588525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.589176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.597168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.597803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.598082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.598357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.599144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:32.599454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.600966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.601360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.603535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.616426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 410/795] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:32.617399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:32.620703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:32.622812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:32.625070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:32.625070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:32.625070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:32.625070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 386/796] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:32.632265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.731635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:32.735646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f60945e-b5d9-4603-8187-3d4a0462da15 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.746223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e24d26d-c560-4261-baae-9cc3abd1735d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.776315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:32.776657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.777358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.777724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.778110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.779651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:32.780031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.781975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.794835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.795753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.796225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.796691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.803802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 411/797] 10.222.0.26 () {56 vars in 1149 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:32.806319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.812715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.814619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:32.831709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:32.832761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:32.833840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:32.833840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:32.833840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:32.836370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 387/798] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:32.838028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:32.928602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fb04dc-0267-425c-b2cc-50825595de2c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:32.969045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:32.969699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:32.969989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:32.970532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:32.971233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:32.971737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:32.973724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:32.996455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 412/799] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:32.996455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.007964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a70dd3-6af9-4cbd-95df-121adbd33831 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.025843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:33.068314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.069396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.070000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.070825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.086492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.089217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:33.108967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:33.110833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:33.112512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:33.112512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:33.112512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:33.116362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 388/800] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:33.118929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.151023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363182ad-355f-4f99-b39c-aef05091fd0d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.162678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:33.199653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.200589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.201141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.201679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.202750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:33.203377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.205364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.221966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 413/801] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:33.224344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.267839 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8484cdaa-7ad0-46a5-899d-1fdc8e58bfa2 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.323825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.324588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.324877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.326059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.334442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.335882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.346323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-de34b15a-cc6d-4d66-80fd-d191861279f8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:33.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:33.349959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 389/802] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c108b6a0-acdd-4a87-b919-cccd6b18c3b3 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.351997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c108b6a0-acdd-4a87-b919-cccd6b18c3b3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.353887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c108b6a0-acdd-4a87-b919-cccd6b18c3b3 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.354941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 390/803] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:33.355694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c108b6a0-acdd-4a87-b919-cccd6b18c3b3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.388311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4187ade-b8f8-43a9-b9ba-95a6f862ac05 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.437467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.438156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.438595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.438985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.439869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:33.440423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.442259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c68d10-dcee-47c8-bd1c-364964826705 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.456089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 414/804] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:33.457662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.457979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.458277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.478408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:33.479958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:33.481059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:33.504719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c108b6a0-acdd-4a87-b919-cccd6b18c3b3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.545357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.545998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.546286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.546940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.558021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.562942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.579803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 415/805] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:33.580639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:33.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:33.601718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 391/806] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:33.603662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.679978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:33.723058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d084492-6bae-4c7f-8e3c-878d06a1dfc1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.745916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c1134d-d6f8-4678-a7f0-bd613cec0f77 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.781162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.782313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.783187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.783311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.784545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:33.784960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.786974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.801094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:33.801948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:33.802092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:33.802249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:33.810161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:33.811954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:33.812107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 416/807] 10.222.0.26 () {56 vars in 1135 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:33.812631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:33.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:33.830463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 392/808] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:33.833623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:33.838591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:33.984812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a860386-bbce-4059-ada0-605f997c6111 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:33.997388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e669201-8cca-4343-af67-632dfa12e0c3 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.012423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.030443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.031200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.031482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.031784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.032917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:34.033461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.036100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.048101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.049083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.049482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.049843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.050684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.053010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:34.063699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:34.072945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 07 00:56:34.073829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:34.074584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 417/809] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:34.074658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 393/810] 10.222.0.26 () {56 vars in 1129 bytes} [Sun Aug 7 00:56:33 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:34.074732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.074824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.173681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.195260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6638fec4-da8f-4bf2-acf0-a944f4aaff92 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.221512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bfb434-628c-4a73-a373-995895a8672a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.240752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.241379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.241654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.241928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.254638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.256120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.260383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.261007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.261401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.261683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.262415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:34.262722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.264171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-3ab8394f-75fd-4a8d-b491-97bde45d22bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:34.271415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:34.273189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:34.274190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:34.274190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:34.274190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:34.274190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 418/811] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:34.274190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8095fc5-19ea-4cb5-b751-b6c49a91d303 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.274190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8095fc5-19ea-4cb5-b751-b6c49a91d303 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.274623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8095fc5-19ea-4cb5-b751-b6c49a91d303 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.275893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 419/812] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:34.280686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8095fc5-19ea-4cb5-b751-b6c49a91d303 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.283517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 394/813] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:34.284449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.326111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.403598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788beffe-a5c2-4411-a8bf-982f2856d00f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.421247 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8095fc5-19ea-4cb5-b751-b6c49a91d303 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.432894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.448567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.449194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.449473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.450379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:34.450684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.452115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.457534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 395/814] 10.222.0.26 () {56 vars in 1124 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:34.458366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.463917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.464786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.465187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.465596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.556892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.558460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.597139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.602580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9595908c-1d62-4d69-ac6d-49d5896ef5fa None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.649837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.650514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.650906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.651186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.651980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:34.652457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.654276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b571adcf-3209-4555-bf7a-fbd895656fae None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.671489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 396/815] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:34.672968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.673263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.673538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.700963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:34.702041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:34.710070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 420/816] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:56:34.711349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.769257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.804953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 397/817] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:34.806295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.857013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4993e89d-a7dd-479b-ae2b-506a72aa2c9b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:34.893187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:34.893849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:34.894286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:34.894575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/4814b0fcdd08486bb0f7ae5d8486b9f9/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:34.897282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:34.918321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=4814b0fcdd08486bb0f7ae5d8486b9f9, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:34.920556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:34.935537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 421/818] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:34 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/4814b0fcdd08486bb0f7ae5d8486b9f9/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:34.936555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:34.959751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee4d64a-af1d-40ab-bff8-e33f0956754e None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.014883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.016089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.016659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.017119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.018435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:35.019024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.022130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.031474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 398/819] 10.222.0.26 () {56 vars in 1124 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:35.032967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.116104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7748406c-4bdc-41fe-95c9-5b419c78baa3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.131797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:35.165159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.166694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.167175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.167645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.175623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.177499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:35.195408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:35.197177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:35.198485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:35.199431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 422/820] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:35.200869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.225463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ed84a2-2ffe-4d1e-a9b2-9fec1276de4d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.271235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:35.292971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.293842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.294249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.294739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.318259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.320429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:35.358899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:35.360040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:35.360040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:35.360040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:35.365795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 399/821] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:35.366867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.390432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c653a5-d888-44b4-ad44-f039b54d0a82 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.456147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.457869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.458159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.458425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] PATH_INFO: `/v3/services/network` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.459017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.461454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:35.471224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:35.472979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:35.475018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 423/822] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:35 2022] GET /identity/v3/services/network => generated 88 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:35.476012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.546016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:35.552927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f3bc7c-9ecd-465e-95ac-8c50efe248b5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.630845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.632487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.633310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.635113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.636634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.640038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.684280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 400/823] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:56:35 2022] POST /identity/v3/endpoints => generated 346 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:35.684280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.690828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001c2d8c-57b7-437c-a029-b123e4961516 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.732052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:35.765804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.767669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.769304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.769890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.770972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:35.771546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.773624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.798200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 424/824] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:35.799745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.881460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5fd200-5c9b-4851-bb75-c65808d4cece None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.944487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:35.945177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:35.945467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:35.945765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:35.947330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:35.947641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:35.949088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:35.965239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 401/825] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:35.967372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:35.975401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15bc542-a0dd-41b3-bcb7-6a56948cefb0 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:35.998105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:36.032185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:36.034445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:36.035300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:36.036005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:36.037098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] RBAC: Adding query filter params (name=network) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:36.037832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:36.041913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:36.051506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 425/826] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:35 2022] GET /identity/v3/services?name=network => generated 124 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:36.052923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:36.090387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:36.196728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c641c8ab-bdbb-4b57-a725-66d056cf3685 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:36.227198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c667cb1e-999c-4c79-ad77-3e90dc367785 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:36.243484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:36.244475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:36.245483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:36.245483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/7342064620f34dabb93dd10f92f444a4/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:36.248355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=7342064620f34dabb93dd10f92f444a4, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:36.250153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:36.296303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 402/827] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:35 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/7342064620f34dabb93dd10f92f444a4/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:36.296303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:36.298071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:36.301647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:36.303872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:36.305004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:36.333775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:36.341423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:36.381633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:36.382788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:36.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:36.385306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 426/828] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:36.395005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:36.478975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:36.512086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a7df00-d75a-4ad9-a492-b343ec6d8038 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:36.558239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:36.559920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:36.560608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:36.561345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:36.562930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] RBAC: Adding query filter params (type=network) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:36.563813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:36.566862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:36.574250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 403/829] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:36 2022] GET /identity/v3/services?type=network => generated 350 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:36.584679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:36.653601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:36.687210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52fc973-7a7d-441d-91fb-c4f031edaaf9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:36.760886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:36.762405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:36.763062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:36.763992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:36.765452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:36.766116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:36.768273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:36.780266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905b9cc8-6c08-4a88-af5e-16652b9f6d72 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:36.806210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 427/830] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 412 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:36.813806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:36.837641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb851a0-5652-4381-acbc-6a15e346bd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:36.837641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb851a0-5652-4381-acbc-6a15e346bd8f None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:36.837641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb851a0-5652-4381-acbc-6a15e346bd8f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:36.837641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb851a0-5652-4381-acbc-6a15e346bd8f None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:36.838555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb851a0-5652-4381-acbc-6a15e346bd8f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:36.838555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb851a0-5652-4381-acbc-6a15e346bd8f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:36.853187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:36.884473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 404/831] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:56:36 2022] POST /identity/v3/endpoints => generated 337 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:37.123664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6890d7-bf76-4193-b757-144761de7c7e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:37.182758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:37.183753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:37.184140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:37.184548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:37.185386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=9f6e6a410517438885907f00fabe80f0) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:37.185800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:37.187484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae7deaa-959d-4a55-a78d-a7a1a69639b4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:37.194697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 428/832] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:36 2022] GET /identity/v3/role_assignments?user.id=9f6e6a410517438885907f00fabe80f0&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:37.522824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d21f6-e479-483f-b9e2-d8194963350d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:37.523586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d21f6-e479-483f-b9e2-d8194963350d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:37.524131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d21f6-e479-483f-b9e2-d8194963350d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:37.526240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 405/833] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:37.532269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:37.532872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:37.533372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:37.554760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:37.557182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:37.559805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:37.639163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 429/834] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:37.663487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d21f6-e479-483f-b9e2-d8194963350d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:37.794552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5d21f6-e479-483f-b9e2-d8194963350d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:37.838334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:37.838994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:37.839279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:37.839620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:37.853841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:37.859633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:37.860631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:56:37.861480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:37.862126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 406/835] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:56:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:37.870296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:38.002121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d7b655-3abb-4ce3-b274-3a2be3d6b21e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:38.041890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:38.042661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.042963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.043283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.043992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:38.044358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:38.045876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:38.052105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:38.056481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 430/836] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:56:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:38.063382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:38.160183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:38.238693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b739d7-4478-438c-8841-17bdae62b33a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:38.293643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:38.294293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.294627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.294891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.312383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:38.312383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-fe79e170-0a81-490c-86c2-a24efa3aedfd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:38.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:38.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:38.330502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:38.330502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:38.330502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:38.330502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 407/837] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:38.335458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:38.459074 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a40196b-673e-454e-b9c1-00b1ab63b522 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.459494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a40196b-673e-454e-b9c1-00b1ab63b522 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.459769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a40196b-673e-454e-b9c1-00b1ab63b522 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.461439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 408/838] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:38.467028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.467410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.467558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.481020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a4afb-5011-4c27-afd6-52cc38c84ea2 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:38.487195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:38.488492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:38.536300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:38.537330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.537832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.538322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.539479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:38.540026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:38.542811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:38.568444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 431/839] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:38.574396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:38.608412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 409/840] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:38 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:38.631509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:38.688711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:38.725212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:38.751014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2deea11-2815-474f-9b28-88cad68348f7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:38.798666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:38.799542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.799826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.800178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.812007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:38.813926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:38.832088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:38.833183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:38.834131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 432/841] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:56:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:38.841115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:38.848012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c401d87-df7d-40df-ba08-bcc37e46c6c6 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:38.914917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:38.916657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:38.917161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:38.917658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:38.918734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:38.921384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:38.946666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:38.949086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:38.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:38.956591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 410/842] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:38 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:38.959744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.001514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb9f623-6dbb-4cf9-8812-0fd440dc3dde None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:39.033681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:39.054916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:39.056573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.057072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.057600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.058826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:39.059377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:39.061631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:39.089300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 433/843] 10.222.0.26 () {56 vars in 1149 bytes} [Sun Aug 7 00:56:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:39.095689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.163114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:39.166771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c825a3ad-56d1-4286-a61f-2764da224a35 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:39.206738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:39.207448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.207845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.208164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.208949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:39.209314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:39.211003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:39.223241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 411/844] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:38 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:39.224641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.291533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3c113-804f-4e3d-9185-d1dd868dfc8e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:39.351668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:39.353206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.353206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.353611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.420784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:39.461289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:39.461885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bb4371-d6b6-4a24-b2fe-c14ee6b7a697 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:39.463514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:39.515257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:39.515877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.516155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.516683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.517314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:39.518024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:39.519755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c54cd9-8c38-40ae-9657-e23b243dff95 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:39.527020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 412/845] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:39 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:39.618297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9ee693-99ce-4203-b70f-6bb89b8846d6 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.619075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9ee693-99ce-4203-b70f-6bb89b8846d6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.619397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9ee693-99ce-4203-b70f-6bb89b8846d6 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.621626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 413/846] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:39.627666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.628102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.628773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.649587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 434/847] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:39 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 07 00:56:39.654578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:39.655441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.656433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:39.684263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:39.742547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 414/848] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:39.756122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:39.763878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.796114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11773bd3-5658-434f-8b6b-f9e86cd39729 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:39.859834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:39.860674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.861251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.861713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.875660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:39.877924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:39.880837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16eb597-5f3e-4599-bf3b-4f35b80c8e9b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:39.903283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:39.904415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:39.905371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:39.905371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:39.905371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:39.911637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 435/849] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:39.914246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.925623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93bf694-f0dd-480b-88b3-34e987b3c73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:39.926691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bf694-f0dd-480b-88b3-34e987b3c73f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.927171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bf694-f0dd-480b-88b3-34e987b3c73f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.927636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bf694-f0dd-480b-88b3-34e987b3c73f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.940214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93bf694-f0dd-480b-88b3-34e987b3c73f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-d93bf694-f0dd-480b-88b3-34e987b3c73f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:39.945626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:39.947191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:56:39.948639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:39.952127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 415/850] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:39.953576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadd0f73-ea88-4e8f-a7dd-569fb19f3f72 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:39.953863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadd0f73-ea88-4e8f-a7dd-569fb19f3f72 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:39.954122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadd0f73-ea88-4e8f-a7dd-569fb19f3f72 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:39.960323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 416/851] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:39.960323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadd0f73-ea88-4e8f-a7dd-569fb19f3f72 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:39.980344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:40.073105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d581608b-852b-4055-b13b-0b001a96c397 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.111123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:40.120079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.121017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.121304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.121579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.122341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:40.122644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.124169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b511d109-7230-4c9e-895c-116cf27e3e45 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.129230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadd0f73-ea88-4e8f-a7dd-569fb19f3f72 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.148288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 436/852] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:40.148288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.148288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.148288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.174564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:40.176028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:40.199710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.200381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.200649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.200925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.201818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:40.202135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.203762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.216282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 417/853] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:40.216651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.287316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 437/854] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:40 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:40.288463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.339397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:40.386483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6582f4a6-00c6-49ed-8790-a8767db5cbb8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.428659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.429309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.429601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.429947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.439051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.440592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.442597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdd5c6a-dcfd-44e8-ac35-6b8aefa14ab4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:40.453935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:40.455136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:40.456335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:40.457710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 418/855] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:40.458693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.472586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:40.491132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.491888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.492168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.492500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.505281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.507795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:40.531552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:40.532801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:40.533889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:40.533889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:40.533889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:40.539513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 438/856] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:40.539513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.596324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8ce642-0270-4034-add6-1e00d056bbc5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.645888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.646684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.647215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.647623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.648629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.650807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:40.665221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:40.667038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:40.669019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 419/857] 10.222.0.26 () {56 vars in 1132 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:40.670159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.676637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:40.681306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcf8536-804a-4a5d-beaf-337bffcabb6f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.738731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.739356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.739662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.740429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.741206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:40.741655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.743585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.760003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 439/858] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:40.760902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.805199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:40.835548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e204ca1-3fe2-4995-93bd-63374b40ef80 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.902831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784c89a-917f-4cc3-931a-39939929d333 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:40.907300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.908205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.908547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.908813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.909786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:40.910222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.911796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.932089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 420/859] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:40.932089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:40.975710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:40.976792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:40.977171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:40.977470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:40.978278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:40.978720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:40.980189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:40.989545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 440/860] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:40.990249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:41.026077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:41.091077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69b6f21-f3dc-4501-9d5b-abf43ad23948 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:41.144951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:41.146035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:41.146649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:41.147106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:41.148675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=4814b0fcdd08486bb0f7ae5d8486b9f9) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:41.149247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:41.152473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:41.164347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 421/861] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/role_assignments?user.id=4814b0fcdd08486bb0f7ae5d8486b9f9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:41.164772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:41.171591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:41.181816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b5b65a-4c94-4845-abf8-57d737518bd4 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:41.238880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:41.239695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:41.240110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:41.240604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:41.255205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:41.257378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:41.277889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:41.279144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:41.280128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 441/862] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:56:40 2022] GET /identity/v3/users/swift => generated 83 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:41.289343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:41.369432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ccf3c6-cd5e-4a96-a1e6-d506d3002465 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:41.411005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:41.423918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:41.424904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:41.425315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:41.425785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:41.427115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:41.427777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:41.430305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:41.442880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 422/863] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:41 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:41.494360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ae9051-fd62-4787-999d-72c614f8ef0e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:41.546870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:41.547677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:41.548137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:41.548477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:41.549405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:41.549720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:41.551318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1d27b-6a86-4ec5-b35c-3d2689f44674 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:41.577487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 442/864] 10.222.0.26 () {56 vars in 1135 bytes} [Sun Aug 7 00:56:41 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:41.583911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:41.761232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856b29dd-73b1-49a1-8212-56167111a75a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:41.816535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:41.817585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:41.818059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:41.818633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:41.917388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:41.933581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:41.935700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:42.009901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10c0fa-c2cf-45e4-89a0-f13af918cb05 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:42.010538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10c0fa-c2cf-45e4-89a0-f13af918cb05 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:42.011012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10c0fa-c2cf-45e4-89a0-f13af918cb05 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:42.016344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 443/865] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:42.019428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:42.019988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:42.020696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:42.044426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:42.046736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:42.068821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:42.100508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 423/866] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:41 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 517 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:56:42.103786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:42.149203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 444/867] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:42 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:42.181235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:42.235104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:42.240783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822a8ea9-8b35-4d5b-822f-75696e9bd2e3 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:42.297880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:42.298641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:42.298991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:42.299333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:42.318886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:42.320793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:42.322117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e7e77-25e4-4b3a-9cde-15a394ea9aa8 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:42.347331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:42.348410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:42.349301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:42.349301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:42.349301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:42.353420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 424/868] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:42.355942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:42.373908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:42.374912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:42.375697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:42.376313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:42.377372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:42.379195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:42.401318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 445/869] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:56:42 2022] POST /identity/v3/services => generated 251 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:56:42.499907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f88aec8-6d4e-45fb-a3f7-0c58c1661911 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:42.550501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:42.551512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:42.552074 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:42.552568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:42.553472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:42.553998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:42.555663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:42.567594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:42.573578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 425/870] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:42.581449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:42.586773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:42.746895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e629bcb-1ee9-4330-8ca9-aeea3e039f10 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:42.789844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:42.791098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:42.791665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:42.792140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:42.800458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:42.802678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:42.817654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:42.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:42.819730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:42.823335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 446/871] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:42.829684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:42.978534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4bf46b-0b80-4858-bfaa-22f6dd0f3ff8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:43.048992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:43.050044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:43.050538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:43.050995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:43.052354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:43.052962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:43.055783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ae8b3c-638d-408c-a95a-6ea6002060a7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:43.081349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 426/872] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:43.088114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:43.177155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:43.222145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e953f0-dcac-4129-ac6d-53f8e25d3e0f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:43.261349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:43.262351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:43.262832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:43.263300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:43.264345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=7342064620f34dabb93dd10f92f444a4) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:43.264884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:43.266897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ed496-9956-499b-af76-91705a267f6f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:43.275465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 447/873] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:43 2022] GET /identity/v3/role_assignments?user.id=7342064620f34dabb93dd10f92f444a4&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:44.143537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77d25b9-6f5b-4769-9d40-bb89ee85b76a None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.144089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77d25b9-6f5b-4769-9d40-bb89ee85b76a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.144089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77d25b9-6f5b-4769-9d40-bb89ee85b76a None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.145822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 427/874] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:44.152367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.152899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.152899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.172165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:44.173408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:44.254725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 448/875] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:44 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:44.282484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:44.283327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c6701-fcab-4a13-9bd1-dc3d774a8540 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.283842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c6701-fcab-4a13-9bd1-dc3d774a8540 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.284352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41c6701-fcab-4a13-9bd1-dc3d774a8540 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.286021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 428/876] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:44.293097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.293707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.294193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.320680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:44.322951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:44.355614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:44.391677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:44.428522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 429/877] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:44 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:44.428791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb68024-78fe-4d59-b9a1-991522662355 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:44.473362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:44.480422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:44.482151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.482805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.483518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.503614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:44.511323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:44.513428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:56:44.514693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:44.518837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 449/878] 10.222.0.26 () {56 vars in 1120 bytes} [Sun Aug 7 00:56:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:44.528294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:44.624814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf41f28-cc4d-4ac9-9886-d930fc4722d7 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:44.653679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfa25b8-527f-4423-bdf7-fe5f7b703369 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:44.675020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:44.677406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:44.679747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.680997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.681690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.683192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:44.685720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:44.698846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:44.699613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.699894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.700368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:44.700927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:44.701230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:44.702833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:44.708865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 430/879] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:56:44 2022] POST /identity/v3/services => generated 269 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:56:44.722061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 450/880] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:56:44 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:44.722272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:44.826102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:44.916564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29868c74-247f-4ebf-868f-7831c0f3aaa0 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:44.975878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:44.976981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:44.977472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:44.977994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.001506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:45.004301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-336f896f-9a9d-4f22-b699-7861d6cc9104 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:45.034249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:45.035251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:45.036118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:45.036118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:45.036118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:45.036824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 431/881] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:45.044806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.183690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799e5c28-25a7-4d41-8fdc-cff5cc47fbce None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:45.220162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec8b74a-e362-408e-b0fb-6d1100b9f484 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.220769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec8b74a-e362-408e-b0fb-6d1100b9f484 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.221053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec8b74a-e362-408e-b0fb-6d1100b9f484 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.223208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 432/882] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:45.230156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.230441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.230614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.250809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:45.251923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.252445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.252914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.254131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:45.254841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:45.257564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:45.262710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:45.264205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:45.286738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 451/883] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:45.289851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.374129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:45.398068 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 433/884] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:45 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:45.450213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.463443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7a4f42-fd97-45f3-a7f4-719962c8343c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:45.483427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:45.532550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:45.534195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.535097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.535819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.552731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:45.556470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:45.573737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26062f00-f4a8-461a-9ac0-2ddd0e029f54 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:45.586660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:45.589022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:45.590890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:45.600750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 452/885] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:45.605861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.617568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:45.618298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.618700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.618972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.619755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:45.621329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:45.632372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:45.634801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:45.636939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 434/886] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:45.640828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.751869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:45.764725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f6c485-a9a8-4709-b30a-9763486d1d4c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:45.813936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:45.814656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.814986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.815347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.816091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:45.816430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:45.817958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:45.828678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 435/887] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:45.830264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.831574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:45.838483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767840f8-52ae-41f9-ab35-c3e94878d67e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:45.886957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:45.888858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.889395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.890039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.891460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:45.892072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:45.895380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970a2a58-2cf2-41a8-9e27-b877924e7a38 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:45.928299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 453/888] 10.222.0.26 () {56 vars in 1149 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:45.930434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11176927-0a4e-4a6e-a3e7-70c2d5851da7 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:45.930967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11176927-0a4e-4a6e-a3e7-70c2d5851da7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:45.931707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11176927-0a4e-4a6e-a3e7-70c2d5851da7 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:45.936333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 454/889] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:45.936333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11176927-0a4e-4a6e-a3e7-70c2d5851da7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:45.998307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f582d090-387d-45d6-bb7f-054c41669e7a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:46.050085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:46.050722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.051295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.051600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.052338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:46.053248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:46.055234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a0edc2-c479-4777-af6f-da921b885ff4 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:46.068377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 436/890] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:46.070900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.071474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.072864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.102533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11176927-0a4e-4a6e-a3e7-70c2d5851da7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:46.105541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:46.107600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:46.130283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:46.157551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:46.158216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.158500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.158961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.193426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:46.236267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 437/891] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:46 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:46.237721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:46.263363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:46.266785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:46.408502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85eb4019-64d6-4871-bace-ef72ebe48e5b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:46.410787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 455/892] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:45 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:56:46.411853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:46.474391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:46.475054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.475363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.475665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.477053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] RBAC: Adding query filter params (interface=public, service_id=75cf2b8bbe74401680fdb75622863ba2, region_id=RegionOne) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:46.477464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:46.479911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:46.498305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 438/893] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:56:46 2022] GET /identity/v3/endpoints?service_id=75cf2b8bbe74401680fdb75622863ba2&interface=public®ion_id=RegionOne => generated 194 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:46.498305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:46.505527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:46.518974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:46.562863 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f4fab-f7ba-4ab7-bd20-df1f2afb2787 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:46.628612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:46.629918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.630245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.630436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.631221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:46.632796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:46.642502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:46.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:46.645330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:46.648894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 456/894] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:46 2022] GET /identity/v3/services/object-store => generated 93 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:46.650501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:46.671584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983c9c06-3336-41c0-bbd6-a83d4bb465f5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:46.739901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:46.740899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.741303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.741709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.761304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:46.763966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:46.787843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:46.789178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:46.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:46.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:46.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:46.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 439/895] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:46.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:46.809305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd4a7eb-b39b-4202-b113-5fbd6019f632 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:46.845633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:46.869265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:46.870303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:46.870789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:46.871260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:46.872395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:46.874783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:46.875133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:46.883827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 457/896] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:56:46 2022] GET /identity/v3/services => generated 1268 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 00:56:46.885243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:46.965081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f42454b-6949-4bfa-827c-ab42adc29272 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:47.011192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:47.012265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.012771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.013281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.014152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:47.014692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:47.016376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:47.028787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 440/897] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:56:46 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:47.030784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:47.035713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f01674-425b-458b-a44a-81ce3ac3b8ea None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:47.080141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:47.080141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.080458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.080556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.081198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:47.081507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:47.083047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:47.111036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 458/898] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:47.120261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:47.220793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:47.238263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03e33f-875c-46be-b47f-d6526f086686 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:47.304559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:47.312589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278cfb7c-e203-4047-aec6-92d7e34890b1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:47.317663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:47.318629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.319077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.319515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.320583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:47.321065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:47.323474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d476de7-f5cf-4d73-ba27-526cd80e6958 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:47.331332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 441/899] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:56:47 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:47.333413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50b0d5-b312-4055-bd78-2a90d84f7550 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.333874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50b0d5-b312-4055-bd78-2a90d84f7550 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.334316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50b0d5-b312-4055-bd78-2a90d84f7550 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.337574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 442/900] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:47.349310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.350310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.351118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.381231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:47.382551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.382947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.383485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.393238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:47.396013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:47.396769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:47.398963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:47.431852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:47.433915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:47.436591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:47.440379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 459/901] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:47.449661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:47.559463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 443/902] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:47 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:47.594166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:47.676413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec442a4c-cdb8-47e8-91b7-ea4243123b34 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:47.690147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:47.747541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:47.748858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.749310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.749832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.750887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:47.751196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:47.752751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:47.776872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 460/903] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:47.778272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313490fa-0951-4c9a-b338-de1b6254bcb5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:47.786057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:47.834986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:47.858089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:47.859184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:47.859689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:47.860173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:47.861015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:47.863686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:47.876778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:47.877833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:47.878816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:47.883077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 444/904] 10.222.0.26 () {56 vars in 1132 bytes} [Sun Aug 7 00:56:47 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:47.884100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:47.988937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec11310-ae62-431a-8e33-3d2fb23b4d4f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:48.027559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0618f5b-5317-4397-8f01-3de4dfa48881 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:48.055206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:48.055815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:48.061344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:48.061977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:48.063290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] RBAC: Adding query filter params (role.id=3f436b5b806b465caaa81c0d0770519f, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=4814b0fcdd08486bb0f7ae5d8486b9f9) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:48.064154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:48.066692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:48.084274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 461/905] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:47 2022] GET /identity/v3/role_assignments?user.id=4814b0fcdd08486bb0f7ae5d8486b9f9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=3f436b5b806b465caaa81c0d0770519f => generated 259 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:48.084477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:48.084477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:48.084477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:48.084477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:48.084777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:48.084777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:48.085757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:48.092499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 445/906] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:47 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:48.098731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:48.138055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:48.263787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da536b-f222-496c-83b3-b3130c30dab9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:48.321088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:48.321741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:48.322022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:48.322333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:48.322916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:48.323212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:48.324631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:48.330148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 462/907] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:48 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:48.340426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:48.426607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:48.478966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f54445-9fac-4b84-bdf7-79582b56a1aa None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:48.515286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:48.515908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:48.516195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:48.516475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:48.517179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] RBAC: Adding query filter params (interface=public, service_id=3e7679a5577f4ac5b6f410834e5d1ad8, region_id=RegionOne) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:48.517479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:48.518887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6b7c00-cf9c-4896-9fbe-c62e1ac1659a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:48.526027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 446/908] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:56:48 2022] GET /identity/v3/endpoints?service_id=3e7679a5577f4ac5b6f410834e5d1ad8&interface=public®ion_id=RegionOne => generated 194 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:48.535823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:48.680397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:48.692986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c408381-8ae2-4992-8a60-523468f78b12 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:48.745466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa5dfb0-8be3-47e7-95c6-8a49ea6159c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:48.746295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa5dfb0-8be3-47e7-95c6-8a49ea6159c7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:48.746724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa5dfb0-8be3-47e7-95c6-8a49ea6159c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:48.747159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa5dfb0-8be3-47e7-95c6-8a49ea6159c7 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:48.747878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa5dfb0-8be3-47e7-95c6-8a49ea6159c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:48.749473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa5dfb0-8be3-47e7-95c6-8a49ea6159c7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:48.756857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 463/909] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:56:48 2022] GET /identity/v3/services => generated 1268 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 00:56:49.156717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5d6d4-54d1-4048-9d53-e9df5a2e7773 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:49.157306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5d6d4-54d1-4048-9d53-e9df5a2e7773 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:49.157766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5d6d4-54d1-4048-9d53-e9df5a2e7773 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:49.159624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 447/910] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:49.165948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:49.166602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:49.167097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:49.189506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:49.191263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:49.287768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 464/911] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:49 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:49.315144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e5d6d4-54d1-4048-9d53-e9df5a2e7773 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:49.447201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e5d6d4-54d1-4048-9d53-e9df5a2e7773 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:49.466221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:49.483791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:49.484477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:49.485093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:49.485373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:49.485973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:49.487365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:49.498845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:49.499919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:49.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 448/912] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:49 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:49.508333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:49.615124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:49.631391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a51454-585e-45cd-a6af-d08b2c50b051 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:49.668306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:49.668925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:49.669341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:49.669540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:49.670163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:49.670464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:49.671910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:49.677373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 465/913] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:49 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:49.684024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:49.806926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03543b38-4138-4e70-94da-99fc1507d356 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:49.843654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:49.844312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:49.844728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:49.845078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:49.845786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:49.846162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:49.847608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f01ecac-4282-492c-960e-51c3e12d6f8e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:49.854166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 449/914] 10.222.0.26 () {56 vars in 1114 bytes} [Sun Aug 7 00:56:49 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:49.859812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:49.982531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a89adab-cb79-4f89-8b02-c6df7cf797c7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:50.037587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba90af6-de3a-4d0b-8af2-d6d601161055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:50.038517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba90af6-de3a-4d0b-8af2-d6d601161055 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.038953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba90af6-de3a-4d0b-8af2-d6d601161055 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.039336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba90af6-de3a-4d0b-8af2-d6d601161055 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.040133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba90af6-de3a-4d0b-8af2-d6d601161055 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:50.042254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba90af6-de3a-4d0b-8af2-d6d601161055 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:50.070209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 466/915] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:56:49 2022] POST /identity/v3/endpoints => generated 346 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:50.167455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c27a876-c17b-4526-b428-999034f40e71 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.167843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c27a876-c17b-4526-b428-999034f40e71 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.168020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c27a876-c17b-4526-b428-999034f40e71 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.169789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 450/916] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:50.182103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f57c0a-d3f5-4dd5-b7d0-cef03e02f1b4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.182472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f57c0a-d3f5-4dd5-b7d0-cef03e02f1b4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.182648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f57c0a-d3f5-4dd5-b7d0-cef03e02f1b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.207431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c9f57c0a-d3f5-4dd5-b7d0-cef03e02f1b4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:50.208880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c9f57c0a-d3f5-4dd5-b7d0-cef03e02f1b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:50.211561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:50.323937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 467/917] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:50 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:50.349535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c27a876-c17b-4526-b428-999034f40e71 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:50.409924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:50.481292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c27a876-c17b-4526-b428-999034f40e71 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:50.519549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff452ef-b2d1-4fb6-8f27-de11699c1ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:50.520617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff452ef-b2d1-4fb6-8f27-de11699c1ed8 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.521117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff452ef-b2d1-4fb6-8f27-de11699c1ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.521640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff452ef-b2d1-4fb6-8f27-de11699c1ed8 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.522479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff452ef-b2d1-4fb6-8f27-de11699c1ed8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:50.524071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff452ef-b2d1-4fb6-8f27-de11699c1ed8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:50.542194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 451/918] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:56:50 2022] POST /identity/v3/services => generated 241 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:50.552549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1276afc-c62d-4e77-9d3a-cec1fcb6a4bd None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.553107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1276afc-c62d-4e77-9d3a-cec1fcb6a4bd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.553107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1276afc-c62d-4e77-9d3a-cec1fcb6a4bd None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.554970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 468/919] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:50.561770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2d4c4-3bcf-4af7-a690-32025103a8e7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.562125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2d4c4-3bcf-4af7-a690-32025103a8e7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.562346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2d4c4-3bcf-4af7-a690-32025103a8e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.583534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-36c2d4c4-3bcf-4af7-a690-32025103a8e7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:50.584711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-36c2d4c4-3bcf-4af7-a690-32025103a8e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:50.684775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 452/920] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:50.707981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1276afc-c62d-4e77-9d3a-cec1fcb6a4bd None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:50.835318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1276afc-c62d-4e77-9d3a-cec1fcb6a4bd None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:50.872737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:50.873383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.873660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.873982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.876137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:50.887581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:50.892438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:50.893404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:50.894367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 469/921] 10.222.0.26 () {56 vars in 1120 bytes} [Sun Aug 7 00:56:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:50.895731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c52b956-9ad4-40d8-ba77-230a9767956d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.900256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c52b956-9ad4-40d8-ba77-230a9767956d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.900256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c52b956-9ad4-40d8-ba77-230a9767956d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.900256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 453/922] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:50.900574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:50.904142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:50.904647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:50.905067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:50.924812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:50.926199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:50.957820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:51.021430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 454/923] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:51.022172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc5486d-166e-40af-983c-bf811766f20b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.045928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.066722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.067336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.067620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.067902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.068601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:51.068906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.070448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.082272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 470/924] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:56:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:51.087821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.159117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9c3a91-944f-4d7a-8201-331f3d36cda8 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.164995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:51.209636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.211165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.212051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.213134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.214587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.217850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.225456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6ec90-c9e8-4404-a707-d24a465cb39b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:51.228466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:51.230123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 07 00:56:51.231786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:51.234697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 455/925] 10.222.0.26 () {56 vars in 1132 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:51.235651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.264684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:51.297934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.299133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.299883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.300417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.326119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.329193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.353874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c63a03-6703-4e45-a3a5-f0e138a5cd5e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:51.357975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:51.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:51.359789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:51.359789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:51.359789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:51.364967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 471/926] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:51.370546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.397956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.398890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.399344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.400404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.401226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:51.401841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.403408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.414373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 456/927] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:51.416284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.516275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:51.531726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad2541a-b8cc-43e0-9710-b4df4fbe5d85 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.571515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.572521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.572632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1d5d5b-5ef8-4167-8ecb-a2d1a2a3cab7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.573080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.573532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.574365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:51.574854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.576435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.586976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 457/928] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:51.588228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.591252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:51.642014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.642976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.643673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.643940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.645121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:51.645639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.648192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.680681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 472/929] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:51.683187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:51.712851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67921950-421e-43ae-b7a9-b66c0a54c428 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.750862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.751856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.752387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.752859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.753743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.755364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.787414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 458/930] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:56:51 2022] POST /identity/v3/endpoints => generated 347 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:51.890320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98fd699-c27f-4dc0-a2ac-fcb3634979a8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:51.951947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:51.953008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:51.953492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:51.953961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:51.963343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:51.965307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:51.979167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:51.980168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:51.981220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:51.983461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 473/931] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:51.992754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:52.005782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:52.119672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edde196-de8d-48ca-862d-f53a33dc4577 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:52.157570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:52.158230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.158511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.158786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.159524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:52.159832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:52.161387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:52.183652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 459/932] 10.222.0.26 () {56 vars in 1149 bytes} [Sun Aug 7 00:56:51 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:52.190287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:52.226620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:52.312713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5c0b32-aa5e-4d28-8a0f-c76533e684cf None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:52.349450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:52.350217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.350428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.350708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.362988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:52.364474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fe589848-c4f3-4a8b-a500-d6617ca2e229 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:52.379996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:52.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:52.382317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:52.382317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:52.382317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:52.382317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 474/933] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:52.390554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:52.465311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:52.485336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00938f57-9203-4603-b5ea-90cf89396305 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.485532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00938f57-9203-4603-b5ea-90cf89396305 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.485790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00938f57-9203-4603-b5ea-90cf89396305 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.488348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 475/934] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:52.492899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.493185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.493989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.511381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301d814b-0697-4123-b70f-1ac65b863cf5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:52.516832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:52.517488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:52.578269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029bd873-fd24-4861-b446-604a43212645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:52.580287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029bd873-fd24-4861-b446-604a43212645 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.581228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029bd873-fd24-4861-b446-604a43212645 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.582090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029bd873-fd24-4861-b446-604a43212645 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.583454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029bd873-fd24-4861-b446-604a43212645 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:52.584394 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029bd873-fd24-4861-b446-604a43212645 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:52.587547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029bd873-fd24-4861-b446-604a43212645 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:52.600220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 476/935] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:52.610399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 460/936] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:52.620357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:52.629019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029bd873-fd24-4861-b446-604a43212645 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:52.637431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:52.780058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff47606-d7b1-4d99-9139-bee16a99cb86 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:52.796209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029bd873-fd24-4861-b446-604a43212645 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:52.825094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:52.826034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.826309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.826581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.831368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:52.834651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:52.836107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:52.842888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:52.843803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.844091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.844396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.845046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:52.846488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:52.849189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING keystone.server.flask.common [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-2f6b74cc-0263-4abb-b377-f18c0c2a4eb1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:52.850635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:52.851626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 477/937] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:52.853193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2dcc6-2412-4981-91de-9515d11f50c4 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.853418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2dcc6-2412-4981-91de-9515d11f50c4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.853805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2dcc6-2412-4981-91de-9515d11f50c4 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.855219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 478/938] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:52.860481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2dcc6-2412-4981-91de-9515d11f50c4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:52.875885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.sql.core [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 07 00:56:52.875885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 07 00:56:52.875885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [parameters: {'id': 'fb825f110de5490497f041f2ff4fbb16', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 07 00:56:52.875885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80178) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-1e3395ed-4ea7-4bca-84c7-30b2817f907b None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id fb825f110de5490497f041f2ff4fbb16.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id fb825f110de5490497f041f2ff4fbb16. Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result = self._query(query) Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application conn.query(q) Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 07 00:56:52.892026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result.read() Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 07 00:56:52.892991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application next(self.gen) Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application yield resource Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application next(self.gen) Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.session.rollback() Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.force_reraise() Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise self.value Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application session.commit() Aug 07 00:56:52.894123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._prepare_impl() Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.session.flush() Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._flush(objects) Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application compat.raise_( Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application flush_context.execute() Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 07 00:56:52.895199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rec.execute(self) Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 07 00:56:52.896119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result = self._query(query) Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application conn.query(q) Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result.read() Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 07 00:56:52.898571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [parameters: {'id': 'fb825f110de5490497f041f2ff4fbb16', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:52.899974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:52.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id fb825f110de5490497f041f2ff4fbb16. Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 461/939] 10.222.0.26 () {60 vars in 1140 bytes} [Sun Aug 7 00:56:52 2022] POST /identity/v3/projects => generated 262 bytes in 265 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:52.903023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:52.905128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:52.924875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:52.927298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:52.952811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:53.020126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2dcc6-2412-4981-91de-9515d11f50c4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.040760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 462/940] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:53.040760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.074821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.075532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.076022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.076280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.076895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:53.077196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.078659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.092531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 479/941] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:53.097732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.160114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478de97c-1de0-4cc6-9301-66faf9f0e956 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.176891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:53.207565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.208600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.208884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.209156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.217380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.219049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.220633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45578716-d2c7-46c4-82e7-ae650dd9d52f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:53.231810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:53.232995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 463/942] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:53.234033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.256693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:53.274023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.280341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.280861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.281348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.282467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.284793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:53.302048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:53.303070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:53.303918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:53.312173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 480/943] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:53.313733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.350580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7267d8-e6c1-4dfb-957a-30b2545a937f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.389351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.390010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.390287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.390561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/4814b0fcdd08486bb0f7ae5d8486b9f9/roles/3f436b5b806b465caaa81c0d0770519f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.392775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=4814b0fcdd08486bb0f7ae5d8486b9f9, role_id=3f436b5b806b465caaa81c0d0770519f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.394219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.409629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 464/944] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:56:53 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/4814b0fcdd08486bb0f7ae5d8486b9f9/roles/3f436b5b806b465caaa81c0d0770519f => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:56:53.410475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.459899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:53.483093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bd2e5f-3619-483b-965d-6936b649ebc3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.529192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.531541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:53.531864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.533609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.564932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 481/945] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:53 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:53.573015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ef12c4-3e15-453d-a736-141f26870a03 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.630736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.632520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.633097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.633616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.635112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:53.635647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.637311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.646543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 465/946] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:56:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:53.657259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.725983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:53.780870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c10f958-3f30-439e-bd97-24441cc79502 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:53.823397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:53.823938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:53.824224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:53.824526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:53.825171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:53.825474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:53.826842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:53.833751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 482/947] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:56:53 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:53.839906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:53.958726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:53.966904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e7ecea-736b-4983-89bd-e5db8e938d72 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:54.004897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:54.005911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.006411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.006866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.007760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] RBAC: Adding query filter params (interface=public, service_id=6a8cb614fca443569a108abb4f5f9491, region_id=RegionOne) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:54.008350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:54.010748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc1a94-31f5-4abd-8829-1264af504a96 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:54.017722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 466/948] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:56:53 2022] GET /identity/v3/endpoints?service_id=6a8cb614fca443569a108abb4f5f9491&interface=public®ion_id=RegionOne => generated 194 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:54.028368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:54.147386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450e029b-5782-474f-b4be-654e4a373491 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:54.157490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaadb91-a1f5-4271-b7a6-0fb29fc6a00c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.157720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaadb91-a1f5-4271-b7a6-0fb29fc6a00c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.157970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baaadb91-a1f5-4271-b7a6-0fb29fc6a00c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.160754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 467/949] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:54.165267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.166055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.166337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.167593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:54.185082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:54.186304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:54.196042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:54.197414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.197858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.198591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.199661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:54.201643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:54.210553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 483/950] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:56:54 2022] GET /identity/v3/services => generated 1497 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 00:56:54.292435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 468/951] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:54.311701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:54.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4896f-021c-4c20-9750-48bab23c3534 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:54.497544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:54.498531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.499053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.499563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] PATH_INFO: `/v3/services/compute` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.500723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:54.503436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:54.521493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:54.522748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:56:54.523677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:54.525095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 484/952] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:54.531612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:54.664663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:54.672953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfa8796-2e59-4fa9-8903-5ac19f2d3f01 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:54.713255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:54.713861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.714148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.714420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.715051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] RBAC: Adding query filter params (name=compute) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:54.715350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:54.716897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2674999a-fe2a-4776-b8a5-2951f6931f2f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:54.723714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 469/953] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:54 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:54.729956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:54.782597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:54.861375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3285591d-3944-4850-bc38-45519e9e04fb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:54.913827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:54.914664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:54.914987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:54.915332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:54.916658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:54.916968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:54.918463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d47fd-4184-44eb-b20c-e98195b10bf4 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:54.925082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 485/954] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:56:54 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:55.598263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7a466-7971-45cb-900a-c66a2ca0de89 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:55.598568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7a466-7971-45cb-900a-c66a2ca0de89 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:55.598863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7a466-7971-45cb-900a-c66a2ca0de89 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:55.600542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 470/955] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:55.607030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:55.607553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:55.607553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:55.627443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:55.628734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:55.713303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 486/956] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:55.741052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7a466-7971-45cb-900a-c66a2ca0de89 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:55.860532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7a466-7971-45cb-900a-c66a2ca0de89 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:55.896911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55488b6-cac5-4e91-bdce-017b1bb39b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:55.897547 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55488b6-cac5-4e91-bdce-017b1bb39b27 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:55.897884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55488b6-cac5-4e91-bdce-017b1bb39b27 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:55.898154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55488b6-cac5-4e91-bdce-017b1bb39b27 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:55.909925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55488b6-cac5-4e91-bdce-017b1bb39b27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-c55488b6-cac5-4e91-bdce-017b1bb39b27 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:56:55.914710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:55.915667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:55.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 471/957] 10.222.0.26 () {56 vars in 1120 bytes} [Sun Aug 7 00:56:55 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:55.922020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:55.961902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.085906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f216a618-a84d-461c-93d2-a197c9e6fe64 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.107143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dfddc5-01cf-4d31-b1ef-1d0095f6ddb1 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.107477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dfddc5-01cf-4d31-b1ef-1d0095f6ddb1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.107694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dfddc5-01cf-4d31-b1ef-1d0095f6ddb1 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.109417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 472/958] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:56.115469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.115658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.115962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.135764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:56.137000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:56.151519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.154259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.155302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.155791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.156291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.157579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:56.158448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.161261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.178308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 487/959] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:56:55 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:56.190197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.211175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 473/960] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:56.236805 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.337478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.343332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166efe5-5b8a-4fd9-9941-3aad800ae012 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.359233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816929b4-bcb4-43a4-89a5-9e55cdee49a4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.390425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.391602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.392330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.392652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.404864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.406212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.406447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.406905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.407373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.407904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.408694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.411240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:56.423766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:56.424838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:56.425739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:56.425739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:56.425739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:56.429935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 488/961] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:56.431677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.433991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.sql.core [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 07 00:56:56.433991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 07 00:56:56.433991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [parameters: {'user_id': 'aad807d9144045e2b2b37204031836a2', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 07 00:56:56.433991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80178) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result = self._query(query) Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application conn.query(q) Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result.read() Aug 07 00:56:56.447540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 07 00:56:56.448521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application next(self.gen) Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application yield resource Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application next(self.gen) Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.session.rollback() Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.force_reraise() Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise self.value Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application session.commit() Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 07 00:56:56.449472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._prepare_impl() Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.session.flush() Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._flush(objects) Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application compat.raise_( Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application flush_context.execute() Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rec.execute(self) Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 07 00:56:56.450454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 07 00:56:56.452339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result = self._query(query) Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application conn.query(q) Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application result.read() Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 07 00:56:56.454719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [parameters: {'user_id': 'aad807d9144045e2b2b37204031836a2', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:56.456973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 07 00:56:56.458946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 474/962] 10.222.0.26 () {60 vars in 1131 bytes} [Sun Aug 7 00:56:56 2022] POST /identity/v3/users => generated 160 bytes in 213 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:56.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.476120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.560858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b44870-cf23-491e-a012-2bccdcce8a8e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.597928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.598599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.598872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.599150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.599848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:56.600147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.601581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6c6328-15ca-4de2-af1f-8f98eff9399d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.601797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafbbfea-6b33-4584-bc06-dee3a7206d3f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.619466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 489/963] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:56.619466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01bb3f-3160-44ef-87f9-7b3fad022e3d None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.619466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01bb3f-3160-44ef-87f9-7b3fad022e3d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.619466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01bb3f-3160-44ef-87f9-7b3fad022e3d None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.622606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 490/964] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:56.623461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01bb3f-3160-44ef-87f9-7b3fad022e3d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.636898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.651265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.651970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.652416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.652697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.661522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.663167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-ed8e0c81-5ae2-4239-8644-4f35728dfc3f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:56.676039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:56.677184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:56:56.678165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 475/965] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/users/cinder => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:56.678765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.678765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.679592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.699045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:56.700720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:56.745399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01bb3f-3160-44ef-87f9-7b3fad022e3d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.772412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.782026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 476/966] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:56.782736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.789196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.790706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.791342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.791625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.803117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.805060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:56:56.818900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:56.819873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:56.820846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 491/967] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:56.822138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.897865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:56.900468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5aa035-ec1f-4af8-baa3-0168a8033a55 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.934632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33cbb79-b80a-4ec8-a5f0-a730a5893c05 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:56.941791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.942653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.942653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.942943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.943642 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:56.943941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.945408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.971358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:56.972046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:56.972356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:56.972633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:56.973218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:56.974618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:56.983475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:56.984581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:56.990657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 492/968] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:56.991453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:56.995220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 477/969] 10.222.0.26 () {56 vars in 1101 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:57.004310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:57.067659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:57.116799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bb840d-032b-4c49-9f4e-7840adb3e343 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.120284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04079c25-4be6-464a-8b0b-a28913ccc9ae None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.160814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.161765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.162216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.162610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.163514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:57.163947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.164413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.165023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.165364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.165642 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.166867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:57.167029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05205b3-913c-45c9-8a4b-f5d0614a9cdd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.167167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.167909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8349c37-c46b-4e26-b193-b2cd2e35368a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.169247 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:57.178503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 478/970] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:57.178765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da825a6-cfab-4b89-a0e8-b9a8d6bdc2ac None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.179003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da825a6-cfab-4b89-a0e8-b9a8d6bdc2ac None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.179330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da825a6-cfab-4b89-a0e8-b9a8d6bdc2ac None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.185848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 479/971] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:57.187054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da825a6-cfab-4b89-a0e8-b9a8d6bdc2ac None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:57.195766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 493/972] 10.222.0.26 () {56 vars in 1149 bytes} [Sun Aug 7 00:56:56 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:57.197650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.198162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.198769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.217918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:57.219511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:57.296616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 494/973] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:57.297467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:57.307290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da825a6-cfab-4b89-a0e8-b9a8d6bdc2ac None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.325550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:57.343828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.344469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.344748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.345020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.345643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:57.345942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.347311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.356972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 480/974] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:56:57 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:57.356972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:57.405866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7063b97c-5971-45e8-b9ea-1f958cfddf85 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.428665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:57.441888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.442426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.442701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.442975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.458258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dafccf-46be-4bc3-b1dd-801181bfacb8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.492681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.493299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.494141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.494409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.495050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.496432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.511335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 481/975] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:56:57 2022] POST /identity/v3/services => generated 242 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:57.513657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:57.514270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.516849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.585944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:57.682942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 495/976] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:56:57 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:56:57.686892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f3254d-cfe7-473e-8cd4-e9f1e34f0a54 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.687568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:57.732771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.733308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.733663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.733930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.734602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.735987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.740591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:57.760286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 482/977] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:56:57 2022] POST /identity/v3/endpoints => generated 349 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:56:57.815108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38634d1-ef0e-4c65-ae5f-91fe9835c692 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:57.875980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:57.876779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:57.877337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:57.877812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:57.903011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:57.905648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:56:57.935472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:57.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:56:57.939290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:56:57.939290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:56:57.939290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:57.939290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 496/978] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:56:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:56:57.945058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:58.074531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:58.087613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50007a55-1d77-4108-8039-733f506eae40 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:58.131885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:58.132777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:58.132777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:58.133309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:58.134509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:58.134822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:58.136382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:58.156835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 483/979] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:56:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:58.158317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:58.237614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:58.275172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e3550a-0eb5-460b-b80b-5140fcf5ecd8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:58.319422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:58.320074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:58.320397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:58.320672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:58.329248 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:58.330661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:56:58.345572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:56:58.346599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:56:58.347625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:56:58.352778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 497/980] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:56:58.354210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:58.471964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911e6ade-c5a4-4773-80f4-b1da6493019b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:58.512538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:58.513137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:58.513416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:58.513690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:58.514494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:58.514807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:58.516292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3a18d-33bb-49e1-b436-abc67ca7441e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:58.533947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 484/981] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:56:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:56:58.535486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:58.661458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf023d08-9491-4ff4-a51a-0bd65b0ec0d1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:58.697564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:58.706454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:56:58.707094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:58.707377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:58.707653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:58.708476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] RBAC: Adding query filter params (role.id=3f436b5b806b465caaa81c0d0770519f, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=4814b0fcdd08486bb0f7ae5d8486b9f9) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:56:58.708788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:56:58.710349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b16d695-5892-49a6-a07a-f831acc82c92 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:56:58.720324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 498/982] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:56:58 2022] GET /identity/v3/role_assignments?user.id=4814b0fcdd08486bb0f7ae5d8486b9f9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=3f436b5b806b465caaa81c0d0770519f => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:56:59.694546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34b414d-5193-4982-bbf1-eaea1d227229 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:59.695051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34b414d-5193-4982-bbf1-eaea1d227229 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:59.695051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34b414d-5193-4982-bbf1-eaea1d227229 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:59.700275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 485/983] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:56:59.702483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b2c08-d6b6-4817-995a-8dc93ae404ad None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:59.702917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b2c08-d6b6-4817-995a-8dc93ae404ad None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:59.703101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100b2c08-d6b6-4817-995a-8dc93ae404ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:59.722898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-100b2c08-d6b6-4817-995a-8dc93ae404ad None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:59.724127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-100b2c08-d6b6-4817-995a-8dc93ae404ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:59.803631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 499/984] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:59 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:56:59.835956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34b414d-5193-4982-bbf1-eaea1d227229 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:56:59.853977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:56:59.866777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eaa1dc-7528-4c7b-8ed0-4127d9030929 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:59.867107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eaa1dc-7528-4c7b-8ed0-4127d9030929 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:59.867376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80eaa1dc-7528-4c7b-8ed0-4127d9030929 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:59.869472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 500/985] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:56:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:56:59.874718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:56:59.875014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:56:59.875285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:56:59.894765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:56:59.896006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:56:59.969384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34b414d-5193-4982-bbf1-eaea1d227229 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:56:59.972430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 501/986] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:56:59 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:56:59.991550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.027021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.027021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.027021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.027361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] PATH_INFO: `/v3/services/compute` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.027795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.029471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:00.040646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:00.041736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:00.043049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 486/987] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:56:59 2022] GET /identity/v3/services/compute => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:00.048123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.080948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.139413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f384ea-0a90-4d0f-95c6-98bb79fa7295 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.156588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.177804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.178311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.178579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.178848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.179453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.180666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fce6b13-008d-4ca9-9b22-c788b77ab42c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.181426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:00.192966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:00.194071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:00.200816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 502/988] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:56:59 2022] GET /identity/v3/services/object-store => generated 93 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:00.202043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.219355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.220203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.220505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.220792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.221446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:00.221757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.223141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.234485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 487/989] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:00.236350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.324297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d063b3af-2445-49d9-b684-4f8d40a6a134 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.344467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.346634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1b1102-c70e-453e-9c52-3e563d42cb35 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.365742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.366381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.366659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.366929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.367559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:00.367858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.369415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.384576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.390141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 503/990] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:00.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:00.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.391169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.405876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 488/991] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:00.413830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.423267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.522192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93443f05-24c7-46da-b80e-14b70939a8cb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.548624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acda56d8-9726-492c-bf20-d41399314430 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.558161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.558775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.559055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.559328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.559964 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:00.560289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.561664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.571957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 504/992] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:00.573435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.589434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.590084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.590358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.590633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.591331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] RBAC: Adding query filter params (interface=public, service_id=0a29c9d001fc4718afce737c94a7f708, region_id=RegionOne) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:00.591633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.593211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.599155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 489/993] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/endpoints?service_id=0a29c9d001fc4718afce737c94a7f708&interface=public®ion_id=RegionOne => generated 194 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:00.607606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.625848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.684479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf152a3e-b4bb-46e8-ba01-ae344d0d425f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.694438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.723099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.723806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.724067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.724289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.725021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] RBAC: Adding query filter params (interface=admin, service_id=6a8cb614fca443569a108abb4f5f9491, region_id=RegionOne) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:00.725354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.726773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.737316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 505/994] 10.222.0.26 () {56 vars in 1249 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/endpoints?service_id=6a8cb614fca443569a108abb4f5f9491&interface=admin®ion_id=RegionOne => generated 193 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:00.737546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ca6a4-c307-469a-beb7-0d9b0d7a0b1d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.738399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:00.782982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c2c37-5741-467a-a90f-2cb517dae076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.783944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c2c37-5741-467a-a90f-2cb517dae076 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.784215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c2c37-5741-467a-a90f-2cb517dae076 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.784536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c2c37-5741-467a-a90f-2cb517dae076 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.785150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8c2c37-5741-467a-a90f-2cb517dae076 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.786554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8c2c37-5741-467a-a90f-2cb517dae076 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.794679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 490/995] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/services => generated 1727 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 00:57:00.869035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb295f9d-3934-4e3c-ad91-05326838498b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:00.883123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:00.892780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b3e68-d07f-4dca-ae09-7f6cdf2ad7c2 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.893345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b3e68-d07f-4dca-ae09-7f6cdf2ad7c2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.893682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b3e68-d07f-4dca-ae09-7f6cdf2ad7c2 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.895093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 491/996] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:00.902008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.902294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.902578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.916721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:00.917366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:00.917649 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:00.917922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:00.918543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:00.920001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:00.922352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:00.923739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:00.927737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 506/997] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:57:00 2022] GET /identity/v3/services => generated 1727 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 00:57:01.025354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 492/998] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:00 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:01.049833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:01.176071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53deae5-d93b-4fac-8d17-cc300356147b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:01.216556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:01.217208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:01.217487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:01.217762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] PATH_INFO: `/v3/services/image` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:01.218357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:01.219742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:01.230336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:01.231596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:01.232462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 507/999] 10.222.0.26 () {56 vars in 1105 bytes} [Sun Aug 7 00:57:01 2022] GET /identity/v3/services/image => generated 86 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:01.238215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:01.328677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:01.370084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8ba64-1720-43a7-bcf8-e25fc4cf7364 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:01.407185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:01.408122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:01.408633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:01.409133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:01.410035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] RBAC: Adding query filter params (name=image) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:01.410541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:01.412124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85af543a-bcc0-4ee4-8850-4a59387ea876 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:01.417623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 493/1000] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:01 2022] GET /identity/v3/services?name=image => generated 122 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:01.423673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:01.469565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:01.553304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72f9da-9fbc-46fa-90f6-733bab85828a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:01.633933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:01.635471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:01.635960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:01.636499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:01.637442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] RBAC: Adding query filter params (type=image) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:01.637986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:01.639718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58518a84-39f9-4085-8523-278cad4e12ce None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:01.646046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 508/1001] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:01 2022] GET /identity/v3/services?type=image => generated 122 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:02.887304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d30ac0-ff70-4fee-b774-5cfca252390d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:02.887985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d30ac0-ff70-4fee-b774-5cfca252390d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:02.887985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d30ac0-ff70-4fee-b774-5cfca252390d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:02.889629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 494/1002] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:02.895426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4fba89-2ab3-482d-b0d0-001b9804f5fe None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:02.895987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4fba89-2ab3-482d-b0d0-001b9804f5fe None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:02.895987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4fba89-2ab3-482d-b0d0-001b9804f5fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:02.915952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5a4fba89-2ab3-482d-b0d0-001b9804f5fe None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:02.917276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5a4fba89-2ab3-482d-b0d0-001b9804f5fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:03.012792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 509/1003] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:02 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:03.036987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2962dce-b8bc-4e2b-b12d-97748e1a807b None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.037271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2962dce-b8bc-4e2b-b12d-97748e1a807b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.037550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2962dce-b8bc-4e2b-b12d-97748e1a807b None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.038899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 495/1004] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:03.045802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.046291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.046768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.050653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2962dce-b8bc-4e2b-b12d-97748e1a807b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.075509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:03.077515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:03.126508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:03.155814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2962dce-b8bc-4e2b-b12d-97748e1a807b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.182823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:03.191725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.192398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.192679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.192949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] PATH_INFO: `/v3/services/compute` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.193538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.194938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.203221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 510/1005] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:03 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:03.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:03.206492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:03.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 496/1006] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:57:03 2022] GET /identity/v3/services/compute => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:03.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.230088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.366498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6867235f-d522-4663-8aa5-295c27c6a52f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.381716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce62abb-d987-4005-b1cf-765102895f1c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.427382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.427923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.428243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.428583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.429344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] RBAC: Adding query filter params (name=compute) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:03.429798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.431919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.440102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.440784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.441070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.441324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.441909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.443291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.449807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 511/1007] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:57:03 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:03.455762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:03.458263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:03.460300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:03.461956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 497/1008] 10.222.0.26 () {56 vars in 1126 bytes} [Sun Aug 7 00:57:03 2022] GET /identity/v3/services/object-store => generated 93 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:03.468131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.504144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:03.552678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:03.605026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95aa90b-f8b7-4742-9a74-b0795f1e0cca None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.605264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd126f-6807-4919-985c-d9bc72b7d97d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.642881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.643519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.643651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.643753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.644030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.644252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.644544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.644721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:03.644823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.644996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.645437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:03.645743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.646727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.647132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.652227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 512/1009] 10.222.0.26 () {56 vars in 1112 bytes} [Sun Aug 7 00:57:03 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:03.652982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 498/1010] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:57:03 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:03.658240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.658411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.779159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:03.794842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b69343f-6f47-4b35-8632-c52a89141c51 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.803736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c72a46a-6ee4-4673-a95a-d7e2f6e3fd25 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:03.835974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9744aaa-c752-4428-9d29-78dc41fc2066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.836625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9744aaa-c752-4428-9d29-78dc41fc2066 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.836939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9744aaa-c752-4428-9d29-78dc41fc2066 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.837723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9744aaa-c752-4428-9d29-78dc41fc2066 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.838389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9744aaa-c752-4428-9d29-78dc41fc2066 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.838727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:03.839770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9744aaa-c752-4428-9d29-78dc41fc2066 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.855821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:03.856630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.856907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.857178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.857891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:03.858195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:03.859608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7482754c-4103-42fb-835c-d160f0793da0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:03.872290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 513/1011] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:57:03 2022] POST /identity/v3/endpoints => generated 334 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:03.872290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a20cf-6f30-45a4-9f20-0cd643db9868 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.872290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a20cf-6f30-45a4-9f20-0cd643db9868 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.872290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a20cf-6f30-45a4-9f20-0cd643db9868 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.872744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 499/1012] 10.222.0.26 () {56 vars in 1122 bytes} [Sun Aug 7 00:57:03 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:03.872842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 514/1013] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:03.872919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:03.872919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:03.872919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:03.875466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a20cf-6f30-45a4-9f20-0cd643db9868 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:03.891701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:03.892980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:04.000435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 500/1014] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:03 2022] POST /identity/v3/auth/tokens => generated 1726 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:04.005371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a20cf-6f30-45a4-9f20-0cd643db9868 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:04.014782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:04.055646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990bdac9-a3c4-42ea-b639-858ce6d965e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:04.056994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990bdac9-a3c4-42ea-b639-858ce6d965e3 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:04.057278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990bdac9-a3c4-42ea-b639-858ce6d965e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:04.057547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990bdac9-a3c4-42ea-b639-858ce6d965e3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:04.058245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990bdac9-a3c4-42ea-b639-858ce6d965e3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:04.059631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990bdac9-a3c4-42ea-b639-858ce6d965e3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:04.079554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 515/1015] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:57:03 2022] POST /identity/v3/endpoints => generated 325 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:57:04.142229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0167d07-8c32-433f-98ea-10132087a3f4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:04.157961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:04.194693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f90c132-9c80-4852-8248-694d104b3be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:04.195798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f90c132-9c80-4852-8248-694d104b3be6 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:04.196880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f90c132-9c80-4852-8248-694d104b3be6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:04.197364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f90c132-9c80-4852-8248-694d104b3be6 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:04.198315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f90c132-9c80-4852-8248-694d104b3be6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:04.199738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f90c132-9c80-4852-8248-694d104b3be6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:04.218401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 501/1016] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:57:04 2022] POST /identity/v3/services => generated 242 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:57:06.095684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fcedc5-8256-41f1-9dea-c26ca57370d5 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.096577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fcedc5-8256-41f1-9dea-c26ca57370d5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.096577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fcedc5-8256-41f1-9dea-c26ca57370d5 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.097734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 516/1017] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:06.103821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.104421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.104421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.124567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:06.125883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:06.222815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de75bd3b-e85b-4f0b-a199-0ced191a0b71 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.223307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de75bd3b-e85b-4f0b-a199-0ced191a0b71 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.223792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de75bd3b-e85b-4f0b-a199-0ced191a0b71 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.232564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 517/1018] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:06 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:06.233811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 502/1019] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:06.239938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c19ff85-428f-4589-93d9-8e5e4e90a3b2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.240953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c19ff85-428f-4589-93d9-8e5e4e90a3b2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.241687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c19ff85-428f-4589-93d9-8e5e4e90a3b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.262550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:06.269563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5c19ff85-428f-4589-93d9-8e5e4e90a3b2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:06.271535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5c19ff85-428f-4589-93d9-8e5e4e90a3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:06.278548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.372688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 518/1020] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:06.374573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce1774a-fd22-4fc9-8008-92ef5a2b51d3 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.375527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce1774a-fd22-4fc9-8008-92ef5a2b51d3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.375527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce1774a-fd22-4fc9-8008-92ef5a2b51d3 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.377062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 519/1021] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:06.383565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.384414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.384783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.395727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:06.408020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:06.409366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:06.415611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76af150-b181-4086-b2e0-8b7fc1e76019 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:06.464159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:06.464850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.465134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.465410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.477509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:06.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:06.483988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 503/1022] 10.222.0.26 () {56 vars in 1120 bytes} [Sun Aug 7 00:57:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:06.484988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.492082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 520/1023] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:06.492988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.578084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:06.610055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582ea36c-8f78-456d-85c1-499ce7756c5c None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:06.633476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7884a085-223b-42f5-bf12-0d9802bf6634 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:06.655294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:06.655953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.656211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.656508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.657743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:06.659202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:06.660898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 504/1024] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:06.661669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.680269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:06.680977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.681587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.681587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.682450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:06.682756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:06.684135 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:06.693565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 521/1025] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:57:06 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:06.694301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.694743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:06.810273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a699b3-a4ac-41e5-8d04-1610aa9d3b28 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:06.824438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bfb240-77e0-4a22-8242-c2e6735022fd None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:06.850306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:06.850951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.851229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.851503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] PATH_INFO: `/v3/services/image` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.852099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:06.853583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:06.863151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:06.864349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:06.865767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:06.874062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:06.874180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 505/1026] 10.222.0.26 () {56 vars in 1105 bytes} [Sun Aug 7 00:57:06 2022] GET /identity/v3/services/image => generated 86 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:06.874180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.876847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:06.877465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:06.877753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:06.878027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:06.878684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:06.880155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:06.915402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 522/1027] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:57:06 2022] POST /identity/v3/projects => generated 311 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:06.916167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:06.993778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:07.017028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc73224d-6458-4e20-bae7-4e06849df877 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.068551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4482e64-8d0a-4215-999f-45384c433107 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.071078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.073685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.073685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.073685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.090015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.091846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:07.118400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:07.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:07.121058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:07.121058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:07.121058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:07.121687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 506/1028] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:07.124975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.125478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.125478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.125698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.126377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] RBAC: Adding query filter params (name=image) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:07.126752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.128287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.129548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.135619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 523/1029] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:06 2022] GET /identity/v3/services?name=image => generated 122 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:07.141915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.206408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:07.288960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84dac0-52ba-4b35-bb29-0d8175712085 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.311815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1e273f-ec54-484a-aa00-3b8045aa2135 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.340755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.341542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.341918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.342268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.342994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] RBAC: Adding query filter params (type=image) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:07.343561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.344842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.352063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 524/1030] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/services?type=image => generated 350 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:07.358772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.363177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.363907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.364185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.364490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.365158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:07.365463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.366869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.370692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:07.393906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 507/1031] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:07.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.511974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43a18a7-1e5f-448b-ae89-5cb76199487d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.526866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:07.548397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677b5b53-c701-4a9e-b3de-917977cf84cc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.565027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.565939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.566528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.567123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.568561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] RBAC: Adding query filter params (interface=public, service_id=4a6ad0f52e6647f0a6f9dce532b7f47b, region_id=RegionOne) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:07.569107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.571143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.577833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 525/1032] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/endpoints?service_id=4a6ad0f52e6647f0a6f9dce532b7f47b&interface=public®ion_id=RegionOne => generated 194 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:07.583288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.596670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.597700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.598151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.598437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.607959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.609784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:57:07.630753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:07.631791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:57:07.632794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:07.636906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 508/1033] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:07.637908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.735992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:07.736657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7009d7-037b-47f4-be77-6a02ce972cf6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.787651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.788305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.788596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.788876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.789540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.793157 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.816274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 526/1034] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/services => generated 1957 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 00:57:07.817053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196659f-b331-4f75-8f3c-64e44ce1f3c8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:07.862706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:07.863356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:07.863631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:07.863907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:07.864723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:07.865114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:07.867434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:07.900259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 509/1035] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:07.904439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:07.953549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:08.051393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2df34-73ea-4757-99d8-19b3e8e0bdb7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:08.121658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:08.123254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:08.123254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:08.123521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:08.242981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:08.244581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:08.308167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:08.396511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 527/1036] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:07 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:08.402754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:08.607040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3474b0-623f-46d1-88e3-1ead43bfeb45 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:08.633481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:08.669381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:08.670347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:08.670636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:08.670932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:08.689043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:08.691294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:08.716705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:08.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:08.721214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:08.721214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:08.721214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:08.722159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 510/1037] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:08.724262 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:08.880980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9a0d77-7c2f-4186-8f3d-9d57740b595e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:08.944777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:08.945814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:08.946852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:08.947335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:08.948489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:08.949026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:08.951573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d071e18-1280-435c-a524-4a83468f0ac8 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:08.956149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:08.975947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 528/1038] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:08.985445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:09.155192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f4c7ed-3bcd-4bbc-a6eb-5b945bfb79d1 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:09.208311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2642dd4-3477-449d-ac38-354fe8badb71 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.208655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2642dd4-3477-449d-ac38-354fe8badb71 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.208942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2642dd4-3477-449d-ac38-354fe8badb71 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.210879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 529/1039] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:09.216391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.216689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.216980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.219595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:09.220661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.221161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.221664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.235568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:09.238030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:09.238222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:09.239493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:57:09.254022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:09.255048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:09.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 511/1040] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:57:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:09.262616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:09.312338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:09.349979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 530/1041] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:09 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:09.376544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:09.422697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db58f6e3-11c3-420d-b09f-debe1be2c6d4 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:09.473663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:09.474710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.474710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.474985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.475616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:09.475928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:09.478461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:09.493978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 512/1042] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:57:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:09.500093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:09.525080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3db9a07-2e06-44f3-8193-5771d01ab3b3 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:09.556565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:09.570250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:09.571244 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.572063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.572367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.573647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:09.576033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:09.578576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 531/1043] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:09.584382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:09.633289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:09.659649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c76879-c5cc-4284-b729-abf01ae4eec7 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:09.724356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:09.724932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.725227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.725501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.726572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] RBAC: Adding query filter params (role.id=3f436b5b806b465caaa81c0d0770519f, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=9f6e6a410517438885907f00fabe80f0) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:09.726956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:09.727094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ba615c-91a6-4855-8cd7-0fc4a81b2722 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:09.730589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93df248f-e92e-40c5-88cf-cb97dddd3005 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:09.737131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 513/1044] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:09 2022] GET /identity/v3/role_assignments?user.id=9f6e6a410517438885907f00fabe80f0&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=3f436b5b806b465caaa81c0d0770519f => generated 259 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:09.797624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c0b097-579f-4200-8952-4dc1fc498ca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:09.798696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c0b097-579f-4200-8952-4dc1fc498ca7 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:09.799201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c0b097-579f-4200-8952-4dc1fc498ca7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:09.799702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c0b097-579f-4200-8952-4dc1fc498ca7 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:09.800958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c0b097-579f-4200-8952-4dc1fc498ca7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:09.803715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c0b097-579f-4200-8952-4dc1fc498ca7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:09.841927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 532/1045] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:57:09 2022] POST /identity/v3/users => generated 285 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:10.839362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ed6513-3e5b-4616-934f-463d3078bf48 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:10.840128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ed6513-3e5b-4616-934f-463d3078bf48 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:10.840567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ed6513-3e5b-4616-934f-463d3078bf48 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:10.841765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 514/1046] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:10.849332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:10.850046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:10.850183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:10.873433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:10.874721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:10.943677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:10.970981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 533/1047] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:10 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:10.996768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ed6513-3e5b-4616-934f-463d3078bf48 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:11.147454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ed6513-3e5b-4616-934f-463d3078bf48 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:11.193868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:11.194901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:11.195372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:11.195843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] PATH_INFO: `/v3/services/image` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:11.197420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:11.199848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:11.217297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:11.218483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 07 00:57:11.219380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:11.224887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 515/1048] 10.222.0.26 () {56 vars in 1105 bytes} [Sun Aug 7 00:57:10 2022] GET /identity/v3/services/image => generated 86 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:11.229316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:11.374688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e178d53-1ab3-4483-b8bf-c6b433d97758 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:11.418491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:11.419153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:11.419432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:11.419701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:11.420655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] RBAC: Adding query filter params (name=image) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:11.420979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:11.422654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:11.429198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 534/1049] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:11 2022] GET /identity/v3/services?name=image => generated 122 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:11.437467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:11.468229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:11.632339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c6c70-a9e7-414c-8e38-a1d565b3ae53 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:11.700876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:11.701893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:11.702396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:11.702887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:11.704099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] RBAC: Adding query filter params (type=image) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:11.704699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:11.707476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce03e909-b876-4c9e-bf37-0fbe4064a9e7 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:11.723592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 516/1050] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:11 2022] GET /identity/v3/services?type=image => generated 350 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:11.725186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:11.820285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:11.899290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bb2463-a6ef-457c-9a00-fdbb32218e4d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:11.945752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:11.947023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:11.947442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:11.947720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:11.948528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:11.950168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:11.952478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc87a4-b84b-4c8f-93e1-f0130f25ae21 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:11.953211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc87a4-b84b-4c8f-93e1-f0130f25ae21 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:11.953358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc87a4-b84b-4c8f-93e1-f0130f25ae21 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:11.955566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 517/1051] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:11.961776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ab86e-4b12-4056-9012-aeac092cbee6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:11.962682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ab86e-4b12-4056-9012-aeac092cbee6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:11.962914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ab86e-4b12-4056-9012-aeac092cbee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:11.988278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 535/1052] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 00:57:11 2022] POST /identity/v3/endpoints => generated 327 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:11.998885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6e6ab86e-4b12-4056-9012-aeac092cbee6 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:12.001470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6e6ab86e-4b12-4056-9012-aeac092cbee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:12.161099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 518/1053] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:12.186050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:12.202204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87d0e21-7689-4bec-a87e-d14945438423 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.202687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87d0e21-7689-4bec-a87e-d14945438423 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.202849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87d0e21-7689-4bec-a87e-d14945438423 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.204622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 519/1054] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:12.211296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.211692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.211938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.221834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:12.237755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:12.239190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:12.342493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 520/1055] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:12.350007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:12.354702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9c5cf1-bf35-4174-90d5-ba073f184227 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:12.394911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:12.418008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:12.418814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.419267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.419667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.434704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:12.439702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:12.440865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:12.442503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 536/1056] 10.222.0.26 () {56 vars in 1120 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:12.450108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:12.568554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c785ba99-647c-48f3-9a8d-074b3cd335c8 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:12.610485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb07cb9b-fb65-485f-962f-d8616760f2f0 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:12.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:12.623079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.623370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.623650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.626064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:12.646164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:12.655985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:12.657552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:12.658407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 521/1057] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:12.664595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:12.664784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:12.666327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.666821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.667318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.668585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:12.669153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:12.671821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:12.687700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 537/1058] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:12.695723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:12.733893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:12.837485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa22139-d7ad-4a27-8595-35744b5c63f0 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:12.870133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129fe7f6-578a-4805-aefe-e6a7ce3ef336 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:12.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:12.902610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.903031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.903492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.904487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:12.904942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:12.906569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:12.925470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 522/1059] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:12.926903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:12.936801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:12.937619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:12.937821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:12.938235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:12.953649 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:12.954846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:12.974886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:12.976039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:12.978462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:12.978462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:12.978462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:12.978462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 538/1060] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:12.983294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:13.021342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:13.101859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823c5c95-f19f-439b-bc2b-546d9e741bf6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.103682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:13.149555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1e522-2a9c-44c2-83fc-2d4964c8aedc None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.153292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:13.153981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:13.154260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:13.154525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] PATH_INFO: `/v3/users/32d4b6be60994b62afd34ae803d36acb` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:13.173655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] RBAC: Authorizing `identity:get_user(user_id=32d4b6be60994b62afd34ae803d36acb)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:13.175897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:13.185561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 523/1061] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/users/32d4b6be60994b62afd34ae803d36acb => generated 285 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:13.186773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:13.198826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:13.199503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:13.199776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:13.200046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:13.200732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:13.201032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:13.202522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:13.225195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 539/1062] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:13.226910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:13.345413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f90a90-32ae-4b57-823f-8539ae7eaff8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.368088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:13.386418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f920f99-499d-4141-8e13-4fa34e99a1b1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.400830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:13.401459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:13.401752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:13.402053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:13.442457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:13.443127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:13.443418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:13.443714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] PATH_INFO: `/v3/users/nova` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:13.446174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:13.454910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:13.459068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:57:13.486282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:13.487455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:13.494480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 540/1063] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:57:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:13.495940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:13.535782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:13.538745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:13.700485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75198b8-8c09-45ad-962d-987d02145dbd None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.716778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 524/1064] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 530 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:13.723187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:13.741286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:13.759677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:13.760516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:13.760951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:13.761243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:13.762332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:13.762864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:13.765753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:13.800801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 541/1065] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:57:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:13.806083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:13.883266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:13.905089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de81eb-c37e-4bd4-94c4-5b381c345de0 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.952782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be9d920-7c04-4a5a-adcd-d0acb610cc30 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:13.973266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:13.974542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:13.975030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:13.975516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:13.977385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] RBAC: Authorizing `identity:get_project(project_id=2a7c87062c0f4994ae4be46d04c5225e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:13.979538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:13.988486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 525/1066] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:13 2022] GET /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e => generated 311 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:13.991835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:14.018374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:14.019175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.019564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.019924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.035529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:14.037428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:14.062862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:14.064923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:14.065962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:14.065962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:14.065962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:14.079272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 542/1067] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:14.079272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:14.142174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:14.173068 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60a389f-7385-4e40-bedc-706168730691 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:14.229304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:14.230266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.230757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.231242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.232667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2a7c87062c0f4994ae4be46d04c5225e, user.id=32d4b6be60994b62afd34ae803d36acb) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:14.233227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:14.236005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:14.247021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 526/1068] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:13 2022] GET /identity/v3/role_assignments?user.id=32d4b6be60994b62afd34ae803d36acb&scope.project.id=2a7c87062c0f4994ae4be46d04c5225e&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:14.265419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc614511-118d-441f-8a76-02a1ccb5f75f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:14.269551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:14.334615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:14.335676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.336180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.336711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.338013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:14.338537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:14.341263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:14.372276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 543/1069] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:14.375781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:14.561974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ebfbe2-8a6a-44a5-8142-7b3550633c0e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:14.605369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:14.606452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.606947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.607414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.616631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:14.618168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-373da97e-a751-40e2-a041-36901b7ce1ef None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:57:14.631989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:14.633042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:14.635504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 527/1070] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:57:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:14.643218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:14.818662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515d585e-ba0c-4557-b37d-ad350bf6bd7d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:14.873182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:14.874072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.874602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.875053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.875980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:14.876512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:14.878386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:14.883343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6c94c-1257-43e7-a4b0-7ad25a3cd3ff None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.883781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6c94c-1257-43e7-a4b0-7ad25a3cd3ff None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.884101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6c94c-1257-43e7-a4b0-7ad25a3cd3ff None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.885776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 528/1071] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:14.894251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:14.894776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:14.895313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:14.903194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 544/1072] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:57:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:14.905137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:14.918651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:14.930367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:14.933843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:14.949898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:15.051640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 529/1073] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:15.092555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d2c2ac-af37-475f-ae77-720b296cab10 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:15.097689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:15.155199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:15.156223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:15.156736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:15.157192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/9f6e6a410517438885907f00fabe80f0/roles/3f436b5b806b465caaa81c0d0770519f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:15.160790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=9f6e6a410517438885907f00fabe80f0, role_id=3f436b5b806b465caaa81c0d0770519f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:15.163283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:15.184294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 545/1074] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:57:14 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/9f6e6a410517438885907f00fabe80f0/roles/3f436b5b806b465caaa81c0d0770519f => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:57:15.264774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90ad56-c068-4968-ae15-9a0c6b04025f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:15.326997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:15.327713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:15.328020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:15.328488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:15.335053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:15.452099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:15.454164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:15.605171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 530/1075] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 07 00:57:15.612139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:15.746112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:15.776414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3ab876-9f97-4dca-8a19-36abf0d522ab None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:15.817360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:15.818041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:15.818308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:15.818619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:15.840483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:15.843250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-1a2d01e0-2968-449a-99f5-36109343f267 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:15.870120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:15.871177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:15.872129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:15.872129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:15.872129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:15.872129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 546/1076] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:15.879792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:15.940013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:16.058929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f9cc2-c565-48e8-af1c-074e1f4749af None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:16.124364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:16.125512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:16.126113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:16.127047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:16.128114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:16.128879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:16.131397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc00bbb8-c273-48aa-a94f-a9c9161fea96 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:16.160285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 531/1077] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:16.737773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d4440-311b-402f-bd77-c161af9fb120 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:16.738325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d4440-311b-402f-bd77-c161af9fb120 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:16.738723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970d4440-311b-402f-bd77-c161af9fb120 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:16.740446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 547/1078] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:16.747674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:16.748449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:16.748449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:16.768842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:16.770138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:16.887361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 532/1079] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:16.917115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d4440-311b-402f-bd77-c161af9fb120 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.040139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970d4440-311b-402f-bd77-c161af9fb120 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:17.091921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:17.092917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.093326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.093802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.100015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:17.114305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:17.121035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:17.122142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:17.123062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 548/1080] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:57:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:17.129859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.163007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:17.249129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1eeb9e-9ef1-455e-87f4-0a38f68c6174 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:17.298155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:17.299001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.299392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.299632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.300430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:17.300736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:17.302456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe2466d-965e-4a6d-b22f-c84bec9faaee None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:17.315010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 533/1081] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:57:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:17.321372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.464385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0522d38-40ff-4268-bd34-b5de1a97e998 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:17.495392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99999f-77ff-42c7-99ab-c7ebcf2a0f6d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.495824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99999f-77ff-42c7-99ab-c7ebcf2a0f6d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.496266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99999f-77ff-42c7-99ab-c7ebcf2a0f6d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.498541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 534/1082] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:17.505008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.505469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.505716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.512351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0357925-0790-4e80-8897-9707757a508d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:17.513074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0357925-0790-4e80-8897-9707757a508d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.513397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0357925-0790-4e80-8897-9707757a508d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.513695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0357925-0790-4e80-8897-9707757a508d None admin] PATH_INFO: `/v3/users/32d4b6be60994b62afd34ae803d36acb` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.515001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0357925-0790-4e80-8897-9707757a508d None admin] RBAC: Authorizing `identity:get_user(user_id=32d4b6be60994b62afd34ae803d36acb)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:17.516543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0357925-0790-4e80-8897-9707757a508d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:17.518296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 549/1083] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:17 2022] GET /identity/v3/users/32d4b6be60994b62afd34ae803d36acb => generated 285 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:17.525005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0357925-0790-4e80-8897-9707757a508d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.533235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:17.534643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:17.614222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:17.660596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 535/1084] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:17.680746 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0357925-0790-4e80-8897-9707757a508d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:17.685357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.706695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:17.742958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:17.743742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.744006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.744377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.745705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] RBAC: Authorizing `identity:get_project(project_id=2a7c87062c0f4994ae4be46d04c5225e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:17.747483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:17.749902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 550/1085] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:17 2022] GET /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e => generated 311 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:17.758428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.846653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b657dbe-533b-42a7-b5af-6c3b770eefdf None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:17.888831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:17.889604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.890016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.890304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.906087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:17.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:17.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:17.915687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 536/1086] 10.222.0.26 () {56 vars in 1120 bytes} [Sun Aug 7 00:57:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:17.922015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:17.937604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542bfc2e-7345-4b9b-b349-1b2063b5a85a None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:17.984411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:17.991884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:17.992594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:17.993637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:17.994124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e/users/32d4b6be60994b62afd34ae803d36acb/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:17.996975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] RBAC: Authorizing `identity:create_grant(project_id=2a7c87062c0f4994ae4be46d04c5225e, user_id=32d4b6be60994b62afd34ae803d36acb, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:17.999214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:18.016041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 551/1087] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:57:17 2022] PUT /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e/users/32d4b6be60994b62afd34ae803d36acb/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:57:18.028155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:18.062746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdb9c45-d11d-4703-a7f7-edf17e28a420 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:18.112187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:18.113260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:18.113763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:18.114262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:18.115727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:18.116264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:18.119535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:18.134393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 537/1088] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:57:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:18.143787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:18.325865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeac717-3b5b-41b3-8758-739c59e60a6f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:18.376313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:18.376959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:18.377241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:18.377541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:18.393347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:18.396132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:18.422520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:18.424734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:18.427004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:18.427004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:18.427004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:18.428508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 552/1089] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:18.431619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:18.592275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b79f6d-5737-40ed-a4fe-80d2241c21ba None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:18.644550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:18.645277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:18.645598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:18.645935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:18.646731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:18.647043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:18.648652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:18.660498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:18.666868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 538/1090] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:18.673893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:18.746062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:18.818260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11280d-feb2-4d8a-b0e5-78101398278f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:18.857796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:18.858428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:18.858703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:18.858975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:18.867691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:18.869756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:57:18.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:18.883979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:18.885888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 553/1091] 10.222.0.26 () {56 vars in 1093 bytes} [Sun Aug 7 00:57:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:18.890990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:19.036405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9908a015-1358-4eee-8bc9-e553b9e817ef None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:19.081563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:19.082186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:19.082466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:19.082736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:19.083658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:19.083970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:19.086064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:19.109816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 539/1092] 10.222.0.26 () {56 vars in 1133 bytes} [Sun Aug 7 00:57:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:19.117461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:19.263651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f96934a-272b-4781-95ba-8549c076f002 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:19.266591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:19.314913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:19.315433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:19.315714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:19.316012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:19.419207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:19.422534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:19.615294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 554/1093] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:19 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 07 00:57:19.623089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:19.740542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:19.799474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a315003-8203-450a-9e15-86c02f9eaf85 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:19.856008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:19.856754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:19.857007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:19.857315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:19.870919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:19.872991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:19.891904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:19.894036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:57:19.895617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:19.895617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:57:19.895617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:19.895617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 540/1094] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:19.903598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:19.907549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:20.077930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e26cd09-2ea3-44d5-8765-158961bc7d69 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:20.126995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:20.127858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.128220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.128595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.129471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:20.129967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:20.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801fc528-5928-4f86-81cc-2e4de22616b2 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:20.149627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 555/1095] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:57:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:20.156675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:20.338525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1bbd39-f7df-44de-9565-bb07de09719e None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.338808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1bbd39-f7df-44de-9565-bb07de09719e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.339095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1bbd39-f7df-44de-9565-bb07de09719e None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.339528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37d50f4-85d5-43d9-9f3b-d422d1a872fe None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:20.340676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 556/1096] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:20.347549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.347549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.347549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.355686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:20.375847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:20.379213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:20.389968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:20.390691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.390965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.391232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.399853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:20.401543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:57:20.419372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:20.421015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:20.422172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 541/1097] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:57:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:20.427801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:20.463345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:20.474459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 557/1098] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:20.499529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:20.562392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5c8623-04a0-40fd-be06-dbb323f9d654 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:20.610872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:20.611659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.612078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.612485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.613496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:20.613967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:20.615925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:20.641408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 542/1099] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:57:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:20.648046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5b339-b6b0-43a2-b985-0db033c8013f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:20.648983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:20.690848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:20.691495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.691771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.692057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.704082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:20.706410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:20.714369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:20.715534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:20.716538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 558/1100] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:57:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:57:20.724003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:20.790970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069120c4-f5b4-4331-a6bb-419a6e304561 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:20.794747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:20.833362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:20.834268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.834721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.835202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.836684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] RBAC: Adding query filter params (role.id=3f436b5b806b465caaa81c0d0770519f, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=9f6e6a410517438885907f00fabe80f0) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:20.837857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:20.840644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:20.849142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 543/1101] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:20 2022] GET /identity/v3/role_assignments?user.id=9f6e6a410517438885907f00fabe80f0&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=3f436b5b806b465caaa81c0d0770519f => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:20.856878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e86f8-691f-40d7-9534-cd5b7b61ff88 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:20.903946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:20.904773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:20.904908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:20.905264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:20.906076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:20.906389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:20.908054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:20.920355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 559/1102] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:57:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:20.926990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:21.059446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:21.070418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c274c2-a3b0-445d-b67f-95f6fd970595 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:21.116069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:21.116768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:21.117044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:21.117318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] PATH_INFO: `/v3/users/32d4b6be60994b62afd34ae803d36acb` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:21.119162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] RBAC: Authorizing `identity:get_user(user_id=32d4b6be60994b62afd34ae803d36acb)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:21.120724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:21.122670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 544/1103] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:20 2022] GET /identity/v3/users/32d4b6be60994b62afd34ae803d36acb => generated 285 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:21.131997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:21.303066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b2e736-b0e7-4034-a5f0-88b90fde089e None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:21.335352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:21.360675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:21.361308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:21.361619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:21.361896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:21.472547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:21.475200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:21.609902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 560/1104] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:21.616764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:21.769685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8516a1b4-2323-44b7-be4f-fbd206f169f9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:21.821103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfddd2e9-ca1f-419a-afdf-6d32a32a435f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:21.821885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddd2e9-ca1f-419a-afdf-6d32a32a435f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:21.822169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddd2e9-ca1f-419a-afdf-6d32a32a435f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:21.822447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfddd2e9-ca1f-419a-afdf-6d32a32a435f None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:21.823807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfddd2e9-ca1f-419a-afdf-6d32a32a435f None admin] RBAC: Authorizing `identity:get_project(project_id=2a7c87062c0f4994ae4be46d04c5225e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:21.825502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfddd2e9-ca1f-419a-afdf-6d32a32a435f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:21.827640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 545/1105] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:21 2022] GET /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e => generated 311 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:21.834112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:21.873386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:21.991742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b4b71-ee24-4deb-89cd-9e89a9b96f98 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:22.033775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:22.034334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:22.034617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:22.035885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:22.035885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2a7c87062c0f4994ae4be46d04c5225e, user.id=32d4b6be60994b62afd34ae803d36acb) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:22.036144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:22.037411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b1d3a68-9004-4acc-973d-b6700992382f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:22.044903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 561/1106] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:21 2022] GET /identity/v3/role_assignments?user.id=32d4b6be60994b62afd34ae803d36acb&scope.project.id=2a7c87062c0f4994ae4be46d04c5225e&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:24.248569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d4f6cc-c8fa-4e65-8236-ba8716acbc31 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:24.249192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d4f6cc-c8fa-4e65-8236-ba8716acbc31 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:24.249192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d4f6cc-c8fa-4e65-8236-ba8716acbc31 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:24.250537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 546/1107] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:24.256800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:24.257362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:24.257362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:24.282500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:24.283928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:24.370907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 562/1108] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:24.395888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d4f6cc-c8fa-4e65-8236-ba8716acbc31 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:24.427109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:24.500997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d4f6cc-c8fa-4e65-8236-ba8716acbc31 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:24.535124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1b6b8f-27f8-4c34-a20c-8aca9027b99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:24.535754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1b6b8f-27f8-4c34-a20c-8aca9027b99c None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:24.536042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1b6b8f-27f8-4c34-a20c-8aca9027b99c None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:24.536343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1b6b8f-27f8-4c34-a20c-8aca9027b99c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:24.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1b6b8f-27f8-4c34-a20c-8aca9027b99c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:24.538910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1b6b8f-27f8-4c34-a20c-8aca9027b99c None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:24.541045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 547/1109] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:24.546907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:24.559366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:24.659564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96186a9-651d-44f4-b277-fcea6d3b5ca8 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:24.697123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa8ef35-b240-4d2c-8f75-6595d90e09d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:24.697733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa8ef35-b240-4d2c-8f75-6595d90e09d0 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:24.698012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa8ef35-b240-4d2c-8f75-6595d90e09d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:24.698284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa8ef35-b240-4d2c-8f75-6595d90e09d0 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:24.698934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa8ef35-b240-4d2c-8f75-6595d90e09d0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:24.700397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa8ef35-b240-4d2c-8f75-6595d90e09d0 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:24.722119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 563/1110] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:57:24 2022] POST /identity/v3/users => generated 286 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:26.818754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13718e-1589-441c-94e7-3f29b4f4f75d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:26.819311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13718e-1589-441c-94e7-3f29b4f4f75d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:26.819311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a13718e-1589-441c-94e7-3f29b4f4f75d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:26.820745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 548/1111] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:26.826774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:26.827368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:26.827368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:26.845861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:26.847072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:26.907456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:26.916997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 564/1112] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:26.942572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13718e-1589-441c-94e7-3f29b4f4f75d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:27.042808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:27.048020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a13718e-1589-441c-94e7-3f29b4f4f75d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:27.081523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:27.082061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:27.082337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:27.082606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] PATH_INFO: `/v3/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:27.083981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] RBAC: Authorizing `identity:get_role(role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:27.085359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:27.086685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 549/1113] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:26 2022] GET /identity/v3/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 223 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:27.093219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:27.192787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94ab316-ad51-467f-8ff2-4b19d4a17131 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:27.227235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:27.227866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:27.228142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:27.228443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] PATH_INFO: `/v3/users/ea1c19b270a44dd687d2bd99b83a47a8` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:27.241754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] RBAC: Authorizing `identity:get_user(user_id=ea1c19b270a44dd687d2bd99b83a47a8)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:27.243694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:27.245378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 565/1114] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:27 2022] GET /identity/v3/users/ea1c19b270a44dd687d2bd99b83a47a8 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:27.251311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:27.355343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6b0ae1-0d34-499d-99b8-f8561860e441 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:27.389161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:27.389765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:27.390044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:27.390315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:27.428316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:27.467691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:27.469209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:27.578585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 550/1115] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:27.585443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:27.651115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:27.705953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b035f0-765c-4707-8342-bd6b689a213b None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:27.744610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6cc4b7-ea49-4609-af86-d95400027bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:27.745277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cc4b7-ea49-4609-af86-d95400027bdd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:27.745559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cc4b7-ea49-4609-af86-d95400027bdd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:27.745837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cc4b7-ea49-4609-af86-d95400027bdd None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:27.746981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6cc4b7-ea49-4609-af86-d95400027bdd None admin] RBAC: Authorizing `identity:get_project(project_id=2a7c87062c0f4994ae4be46d04c5225e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:27.748449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6cc4b7-ea49-4609-af86-d95400027bdd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:27.750109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 566/1116] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:27 2022] GET /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e => generated 311 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:27.756319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:27.823453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:27.857228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f1f5c3-5b5a-4983-97b2-10332e76ee23 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:27.891470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:27.892074 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:27.892380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:27.892654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:27.893368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] RBAC: Adding query filter params (role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=2a7c87062c0f4994ae4be46d04c5225e, user.id=ea1c19b270a44dd687d2bd99b83a47a8) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:27.893672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:27.895099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb6d3f3-b3f9-496e-a12e-1d5fcc7f1888 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:27.901336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 551/1117] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:27 2022] GET /identity/v3/role_assignments?user.id=ea1c19b270a44dd687d2bd99b83a47a8&scope.project.id=2a7c87062c0f4994ae4be46d04c5225e&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:29.946982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb9c12-4e17-478f-a556-b9ada704ca44 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:29.947539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb9c12-4e17-478f-a556-b9ada704ca44 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:29.947539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbb9c12-4e17-478f-a556-b9ada704ca44 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:29.948912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 567/1118] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:29.954560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:29.955070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:29.955195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:29.973475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:29.974665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:30.043043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 552/1119] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:30.065126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbb9c12-4e17-478f-a556-b9ada704ca44 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:30.164225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbb9c12-4e17-478f-a556-b9ada704ca44 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:30.188295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:30.200389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:30.200954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:30.201234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:30.201506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] PATH_INFO: `/v3/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:30.202874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] RBAC: Authorizing `identity:get_role(role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:30.204252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:30.205565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 568/1120] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:30 2022] GET /identity/v3/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:30.211462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:30.270785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:30.314835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439e0dee-07fb-41f1-a86c-610313e2e5e4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:30.348176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:30.348842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:30.349120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:30.349392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] PATH_INFO: `/v3/users/ea1c19b270a44dd687d2bd99b83a47a8` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:30.350500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] RBAC: Authorizing `identity:get_user(user_id=ea1c19b270a44dd687d2bd99b83a47a8)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:30.351940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:30.354133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 553/1121] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:30 2022] GET /identity/v3/users/ea1c19b270a44dd687d2bd99b83a47a8 => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:30.359957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:30.459393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a378658-0aa8-4599-92e7-daa01db58010 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:30.495375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bf5afc-6f17-4297-9ce2-29129120adc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:30.496005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf5afc-6f17-4297-9ce2-29129120adc0 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:30.496311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf5afc-6f17-4297-9ce2-29129120adc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:30.496586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf5afc-6f17-4297-9ce2-29129120adc0 None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:30.497692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bf5afc-6f17-4297-9ce2-29129120adc0 None admin] RBAC: Authorizing `identity:get_project(project_id=2a7c87062c0f4994ae4be46d04c5225e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:30.499113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bf5afc-6f17-4297-9ce2-29129120adc0 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:30.500768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 569/1122] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:30 2022] GET /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:30.506850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:30.605262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de318bd8-6823-42ca-b2a9-3bfbe2c6d95e None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:30.635818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:30.646198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468ac660-c58e-4946-be10-e04a83cfed79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:30.647142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ac660-c58e-4946-be10-e04a83cfed79 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:30.647501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ac660-c58e-4946-be10-e04a83cfed79 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:30.647934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ac660-c58e-4946-be10-e04a83cfed79 None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e/users/ea1c19b270a44dd687d2bd99b83a47a8/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:30.650874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ac660-c58e-4946-be10-e04a83cfed79 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a7c87062c0f4994ae4be46d04c5225e, user_id=ea1c19b270a44dd687d2bd99b83a47a8, role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:30.653069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ac660-c58e-4946-be10-e04a83cfed79 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:30.670132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 554/1123] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:57:30 2022] PUT /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e/users/ea1c19b270a44dd687d2bd99b83a47a8/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:57:32.718622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7d869-e138-4223-86c1-3d432e618722 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:32.719162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7d869-e138-4223-86c1-3d432e618722 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:32.719289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7d869-e138-4223-86c1-3d432e618722 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:32.720972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 570/1124] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:57:32.726513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:32.727818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:32.727818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:32.745736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:32.746953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:32.820396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 555/1125] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:32.845522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7d869-e138-4223-86c1-3d432e618722 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:32.911258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:32.961763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7d869-e138-4223-86c1-3d432e618722 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:32.998681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:32.999230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:32.999511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:32.999784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] PATH_INFO: `/v3/roles/6b27548a5bc24a3fa9c938bf459ffa14` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:33.001231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] RBAC: Authorizing `identity:get_role(role_id=6b27548a5bc24a3fa9c938bf459ffa14)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:33.002594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:33.003923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 571/1126] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:32 2022] GET /identity/v3/roles/6b27548a5bc24a3fa9c938bf459ffa14 => generated 223 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:33.011331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:33.132958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa16f37-331d-4de2-ad96-20988b284d5c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:33.151826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:33.170494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:33.171136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:33.171416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:33.171689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] PATH_INFO: `/v3/users/ea1c19b270a44dd687d2bd99b83a47a8` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:33.172864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] RBAC: Authorizing `identity:get_user(user_id=ea1c19b270a44dd687d2bd99b83a47a8)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:33.174304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:33.175914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 556/1127] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:33 2022] GET /identity/v3/users/ea1c19b270a44dd687d2bd99b83a47a8 => generated 286 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:33.182940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:33.295305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1fb656-5d2c-4db5-a62f-7dc44de89161 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:33.323141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:33.334156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:33.335133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:33.335654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:33.335937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:33.412574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:33.414043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:33.516074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 572/1128] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:33.522703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:33.640514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc06df06-e33e-4e14-84c3-3f88f2fce9b8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:33.677304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e67799-e5dc-4d29-acb3-287212e7a394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:33.677938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e67799-e5dc-4d29-acb3-287212e7a394 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:33.678213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e67799-e5dc-4d29-acb3-287212e7a394 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:33.678483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e67799-e5dc-4d29-acb3-287212e7a394 None admin] PATH_INFO: `/v3/projects/2a7c87062c0f4994ae4be46d04c5225e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:33.679616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e67799-e5dc-4d29-acb3-287212e7a394 None admin] RBAC: Authorizing `identity:get_project(project_id=2a7c87062c0f4994ae4be46d04c5225e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:33.681725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e67799-e5dc-4d29-acb3-287212e7a394 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:33.683647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 557/1129] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:33 2022] GET /identity/v3/projects/2a7c87062c0f4994ae4be46d04c5225e => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:33.690694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:33.721498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:33.801432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03779844-3dea-4bec-837b-b82b330ed8bf None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:33.836031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:33.836665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:33.837566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:33.837778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:33.838499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] RBAC: Adding query filter params (role.id=6b27548a5bc24a3fa9c938bf459ffa14, scope.project.id=2a7c87062c0f4994ae4be46d04c5225e, user.id=ea1c19b270a44dd687d2bd99b83a47a8) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:33.838806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:33.840228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab7f9c1-45ba-468a-9194-1b6076f8d9d2 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:33.846268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 573/1130] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:33 2022] GET /identity/v3/role_assignments?user.id=ea1c19b270a44dd687d2bd99b83a47a8&scope.project.id=2a7c87062c0f4994ae4be46d04c5225e&role.id=6b27548a5bc24a3fa9c938bf459ffa14 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:35.954370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e069a3-9160-4632-afcb-b4913fc719ae None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:35.954921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e069a3-9160-4632-afcb-b4913fc719ae None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:35.954921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e069a3-9160-4632-afcb-b4913fc719ae None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:35.956284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 558/1131] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:35.961424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:35.962299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:35.962507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:35.981229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:35.982409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:36.049104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 574/1132] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:36.072639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e069a3-9160-4632-afcb-b4913fc719ae None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:36.137543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:36.186854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e069a3-9160-4632-afcb-b4913fc719ae None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:36.221961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a4dc9-c310-42ab-9e16-b67fcd10924f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:36.223268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a4dc9-c310-42ab-9e16-b67fcd10924f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:36.223268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a4dc9-c310-42ab-9e16-b67fcd10924f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:36.223268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a4dc9-c310-42ab-9e16-b67fcd10924f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:36.224408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3a4dc9-c310-42ab-9e16-b67fcd10924f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:36.225632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3a4dc9-c310-42ab-9e16-b67fcd10924f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:36.227218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 559/1133] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:36.238421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:36.260845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:36.343072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d2b752-77b9-40c6-8747-96c5c4c7735a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:36.378656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfded55b-76f7-46d8-a59a-2aeccfae3b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:36.379263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfded55b-76f7-46d8-a59a-2aeccfae3b98 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:36.379542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfded55b-76f7-46d8-a59a-2aeccfae3b98 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:36.379814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfded55b-76f7-46d8-a59a-2aeccfae3b98 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:36.380485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfded55b-76f7-46d8-a59a-2aeccfae3b98 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:36.381919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfded55b-76f7-46d8-a59a-2aeccfae3b98 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:36.416283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 575/1134] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:57:36 2022] POST /identity/v3/projects => generated 311 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:38.479984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8445aa-2cad-4e0b-a033-a8aca7584151 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:38.480581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8445aa-2cad-4e0b-a033-a8aca7584151 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:38.480581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8445aa-2cad-4e0b-a033-a8aca7584151 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:38.481907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 560/1135] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:38.487493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:38.487942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:38.488056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:38.506392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:38.507592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:38.574516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 576/1136] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:38.598725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8445aa-2cad-4e0b-a033-a8aca7584151 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:38.702950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8445aa-2cad-4e0b-a033-a8aca7584151 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:38.726763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:38.739484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef6fb5d-7ec9-4dfc-bd51-ba7618e98b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:38.740134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef6fb5d-7ec9-4dfc-bd51-ba7618e98b27 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:38.740448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef6fb5d-7ec9-4dfc-bd51-ba7618e98b27 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:38.740722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef6fb5d-7ec9-4dfc-bd51-ba7618e98b27 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:38.741900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef6fb5d-7ec9-4dfc-bd51-ba7618e98b27 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:38.743926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef6fb5d-7ec9-4dfc-bd51-ba7618e98b27 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:38.745643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 561/1137] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:57:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:38.751812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:38.754322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:38.855898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21159712-0903-4384-af47-b7c619a9fe20 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:38.889404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17478c29-c0c0-48c6-90cf-a560ac63d797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:38.890012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17478c29-c0c0-48c6-90cf-a560ac63d797 None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:38.890290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17478c29-c0c0-48c6-90cf-a560ac63d797 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:38.890559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17478c29-c0c0-48c6-90cf-a560ac63d797 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:38.891215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17478c29-c0c0-48c6-90cf-a560ac63d797 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:38.892683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17478c29-c0c0-48c6-90cf-a560ac63d797 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:38.914748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 577/1138] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:57:38 2022] POST /identity/v3/users => generated 286 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 00:57:41.004319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9bec59-a791-4c8d-a62a-1edafe6acfea None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:41.004888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9bec59-a791-4c8d-a62a-1edafe6acfea None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:41.004888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9bec59-a791-4c8d-a62a-1edafe6acfea None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:41.006205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 562/1139] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:41.012335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:41.012856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:41.012958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:41.031532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:41.032779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:41.087065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:41.102967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 578/1140] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:41.128767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9bec59-a791-4c8d-a62a-1edafe6acfea None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:41.236807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9bec59-a791-4c8d-a62a-1edafe6acfea None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:41.272725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:41.273719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:41.274176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:41.275134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:41.293304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:41.301995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:41.303107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:41.304083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 563/1141] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:57:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:41.310414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:41.426372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd207c36-0e54-4b2b-9f1c-80aa5f18a39d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:41.460937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:41.461543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:41.462346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:41.462619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:41.463282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:41.463585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:41.464977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:41.473165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 579/1142] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:57:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:41.479631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:41.530659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:41.584069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2921df86-9648-4a60-926e-5f42f7fd528a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:41.622732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:41.623371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:41.623651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:41.623924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] PATH_INFO: `/v3/users/d0ad19683e094e598e2d42474ba3185a` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:41.638095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] RBAC: Authorizing `identity:get_user(user_id=d0ad19683e094e598e2d42474ba3185a)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:41.639706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:41.641432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 564/1143] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:41 2022] GET /identity/v3/users/d0ad19683e094e598e2d42474ba3185a => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:41.647379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:41.701832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:41.755285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1752bf29-5525-467c-b870-921ad8d72d4d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:41.790893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:41.791409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:41.791687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:41.791955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:41.858174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:41.859617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:41.906140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:41.954936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 580/1144] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 07 00:57:41.960857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:42.060829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af18e30c-f64d-4f72-9be3-f5c795c3c3a3 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:42.069856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:42.096159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee72091-ac98-4f7f-9b56-f60771bcd7cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:42.096845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee72091-ac98-4f7f-9b56-f60771bcd7cf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:42.097127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee72091-ac98-4f7f-9b56-f60771bcd7cf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:42.097402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee72091-ac98-4f7f-9b56-f60771bcd7cf None admin] PATH_INFO: `/v3/projects/f85b7600d94f47bba4a6978abcc609a3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:42.098600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee72091-ac98-4f7f-9b56-f60771bcd7cf None admin] RBAC: Authorizing `identity:get_project(project_id=f85b7600d94f47bba4a6978abcc609a3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:42.100055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee72091-ac98-4f7f-9b56-f60771bcd7cf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:42.101786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 565/1145] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:41 2022] GET /identity/v3/projects/f85b7600d94f47bba4a6978abcc609a3 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:42.109273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:42.230888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c61193c-cbcc-4bed-9a55-d40903ef6b59 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:42.263910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:42.264549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:42.264833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:42.265105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:42.265821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=f85b7600d94f47bba4a6978abcc609a3, user.id=d0ad19683e094e598e2d42474ba3185a) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:42.266128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:42.267563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3befa7c1-4f15-482d-b475-928b4346f233 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:42.272860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 581/1146] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:42 2022] GET /identity/v3/role_assignments?user.id=d0ad19683e094e598e2d42474ba3185a&scope.project.id=f85b7600d94f47bba4a6978abcc609a3&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:44.246990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e88b82-b0bb-4332-9e71-08ef216b6904 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:44.247528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e88b82-b0bb-4332-9e71-08ef216b6904 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:44.247528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e88b82-b0bb-4332-9e71-08ef216b6904 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:44.248958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 566/1147] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:44.254790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:44.255250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:44.255359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:44.273871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:44.275075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:44.284716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:44.342562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 582/1148] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:44.364780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e88b82-b0bb-4332-9e71-08ef216b6904 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:44.464381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e88b82-b0bb-4332-9e71-08ef216b6904 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:44.496640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:44.497305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:44.497530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:44.497798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:44.507375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:44.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:44.513723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:44.514977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 567/1149] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:57:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:44.519714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:44.615412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeab068-98d6-4720-ba28-9f57730c3298 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:44.648282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:44.648884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:44.649165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:44.650040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:44.650704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:44.651034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:44.652421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:44.662755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 583/1150] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:57:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:44.668737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:44.704825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:44.769530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fc3b5-3936-403d-92ff-c2fef1b09643 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:44.804218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:44.804883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:44.805164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:44.805439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] PATH_INFO: `/v3/users/d0ad19683e094e598e2d42474ba3185a` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:44.806592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] RBAC: Authorizing `identity:get_user(user_id=d0ad19683e094e598e2d42474ba3185a)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:44.808303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:44.809907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 568/1151] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:44 2022] GET /identity/v3/users/d0ad19683e094e598e2d42474ba3185a => generated 286 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:44.816007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:44.825891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:44.914285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a115205-50f3-44d9-9bf2-77397389f01c None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:44.947879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10586811-7117-4933-b667-87438b4d9c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:44.948573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10586811-7117-4933-b667-87438b4d9c39 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:44.948890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10586811-7117-4933-b667-87438b4d9c39 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:44.949199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10586811-7117-4933-b667-87438b4d9c39 None admin] PATH_INFO: `/v3/projects/f85b7600d94f47bba4a6978abcc609a3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:44.950369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10586811-7117-4933-b667-87438b4d9c39 None admin] RBAC: Authorizing `identity:get_project(project_id=f85b7600d94f47bba4a6978abcc609a3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:44.951935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10586811-7117-4933-b667-87438b4d9c39 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:44.953603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 584/1152] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:44 2022] GET /identity/v3/projects/f85b7600d94f47bba4a6978abcc609a3 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:44.960051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:45.055161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0269f44-6a1b-4fef-aaa4-7a7bb63f33b2 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:45.057870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:45.087945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f8e1ab-1a37-4386-b25f-cb63da8aa2b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:45.088619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f8e1ab-1a37-4386-b25f-cb63da8aa2b3 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:45.088895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f8e1ab-1a37-4386-b25f-cb63da8aa2b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:45.089164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f8e1ab-1a37-4386-b25f-cb63da8aa2b3 None admin] PATH_INFO: `/v3/projects/f85b7600d94f47bba4a6978abcc609a3/users/d0ad19683e094e598e2d42474ba3185a/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:45.091007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f8e1ab-1a37-4386-b25f-cb63da8aa2b3 None admin] RBAC: Authorizing `identity:create_grant(project_id=f85b7600d94f47bba4a6978abcc609a3, user_id=d0ad19683e094e598e2d42474ba3185a, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:45.092448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f8e1ab-1a37-4386-b25f-cb63da8aa2b3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:45.105040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 569/1153] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:57:44 2022] PUT /identity/v3/projects/f85b7600d94f47bba4a6978abcc609a3/users/d0ad19683e094e598e2d42474ba3185a/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:57:47.098366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a692d2-9eac-49ed-8e50-f97462c82628 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:47.098900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a692d2-9eac-49ed-8e50-f97462c82628 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:47.098900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a692d2-9eac-49ed-8e50-f97462c82628 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:47.100217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 585/1154] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:47.105956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:47.106364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:47.106506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:47.125071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:47.126258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:47.190976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 570/1155] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:47.214883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a692d2-9eac-49ed-8e50-f97462c82628 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:47.290599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:47.325674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a692d2-9eac-49ed-8e50-f97462c82628 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:47.359931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:47.360553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:47.360826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:47.361095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:47.372186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:57:47.377081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:47.379748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:57:47.381674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 586/1156] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:57:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:47.386628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:47.500959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:47.509350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3328a584-77b7-40a0-befa-74adb5e608dd None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:47.544604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:47.545210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:47.545486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:47.545752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:47.546402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:47.546698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:47.548047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:47.557154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 571/1157] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:57:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:47.563208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:47.665301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de3778b-3281-4430-842d-0a8ad33afeb9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:47.699949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:47.700602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:47.700886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:47.701162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] PATH_INFO: `/v3/users/d0ad19683e094e598e2d42474ba3185a` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:47.702289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] RBAC: Authorizing `identity:get_user(user_id=d0ad19683e094e598e2d42474ba3185a)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:47.703761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:47.705382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 587/1158] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:57:47 2022] GET /identity/v3/users/d0ad19683e094e598e2d42474ba3185a => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:47.711420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:47.813211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15a209b-e895-4db3-b3e1-072f3122b550 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:47.846862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:47.847479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:47.847761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:47.848034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:47.874265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:47.918621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:47.920002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:48.030077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 572/1159] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:47 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:48.036083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:48.038962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:48.136496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bf6887-d5dc-4f77-b1f6-3f51090cbc63 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:48.170364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300b69e8-868b-4836-b14f-515ffde7af42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:48.170909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b69e8-868b-4836-b14f-515ffde7af42 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:48.171191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b69e8-868b-4836-b14f-515ffde7af42 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:48.171467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b69e8-868b-4836-b14f-515ffde7af42 None admin] PATH_INFO: `/v3/projects/f85b7600d94f47bba4a6978abcc609a3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:48.172645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300b69e8-868b-4836-b14f-515ffde7af42 None admin] RBAC: Authorizing `identity:get_project(project_id=f85b7600d94f47bba4a6978abcc609a3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:48.174071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300b69e8-868b-4836-b14f-515ffde7af42 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:48.175692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 588/1160] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:57:48 2022] GET /identity/v3/projects/f85b7600d94f47bba4a6978abcc609a3 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:48.181912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:48.236351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:48.279339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76dae9-dcb9-44d4-b1f9-4f1490c7d512 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:48.323031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:48.324284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:48.324562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:48.324826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:48.325605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=f85b7600d94f47bba4a6978abcc609a3, user.id=d0ad19683e094e598e2d42474ba3185a) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:48.325907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:48.327968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c846950e-f207-4450-be42-af4b85559bcd None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:48.334605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 573/1161] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:57:48 2022] GET /identity/v3/role_assignments?user.id=d0ad19683e094e598e2d42474ba3185a&scope.project.id=f85b7600d94f47bba4a6978abcc609a3&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:50.394349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42424b9-ea53-4afa-9527-4252231b6a87 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:50.394875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42424b9-ea53-4afa-9527-4252231b6a87 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:50.394875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42424b9-ea53-4afa-9527-4252231b6a87 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:50.396195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 589/1162] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:50.401457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:50.401908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:50.402011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:50.421521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:50.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:50.493672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 574/1163] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:50.516265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42424b9-ea53-4afa-9527-4252231b6a87 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:50.585365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:50.616607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42424b9-ea53-4afa-9527-4252231b6a87 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:50.652910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:50.653510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:50.653786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:50.654057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:50.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:50.720968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:50.785071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:50.816730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 590/1164] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:57:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:57:50.823283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:50.871796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:50.923238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79696e38-84d5-49c8-b7cc-25efd331e489 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:50.966049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb5495-6366-4489-b395-475b090db786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:50.966685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb5495-6366-4489-b395-475b090db786 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:50.966962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb5495-6366-4489-b395-475b090db786 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:50.967231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb5495-6366-4489-b395-475b090db786 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:50.973812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fb5495-6366-4489-b395-475b090db786 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:50.975238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fb5495-6366-4489-b395-475b090db786 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-25fb5495-6366-4489-b395-475b090db786 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:57:50.986230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:57:50.987536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:57:50.988729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 575/1165] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:57:50 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:57:50.994047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:51.092984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee7068d-c4d4-4b79-8f87-3ea51f2b2984 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:51.127526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:51.128041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:51.128354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:51.128630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:51.129267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:57:51.129565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:51.130925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d6147-8491-4317-b4e3-85b2e2054009 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:51.138487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 591/1166] 10.222.0.26 () {56 vars in 1115 bytes} [Sun Aug 7 00:57:50 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:53.179725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb90aef-de30-4e19-b8ce-992004f90442 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:53.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb90aef-de30-4e19-b8ce-992004f90442 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:53.180287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb90aef-de30-4e19-b8ce-992004f90442 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:53.181635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 576/1167] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:53.187515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38835f5a-4d27-4903-9bf1-19207a120c59 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:53.187936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38835f5a-4d27-4903-9bf1-19207a120c59 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:53.188067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38835f5a-4d27-4903-9bf1-19207a120c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:53.207695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-38835f5a-4d27-4903-9bf1-19207a120c59 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:53.208751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-38835f5a-4d27-4903-9bf1-19207a120c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:53.237998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:53.282480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 592/1168] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:53.305366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb90aef-de30-4e19-b8ce-992004f90442 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:53.438623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb90aef-de30-4e19-b8ce-992004f90442 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:53.450750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4460e-db92-4c79-be80-9d39021ca609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:53.474969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4460e-db92-4c79-be80-9d39021ca609 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:53.475246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4460e-db92-4c79-be80-9d39021ca609 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:53.475515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4460e-db92-4c79-be80-9d39021ca609 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:53.476180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b4460e-db92-4c79-be80-9d39021ca609 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:53.477621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b4460e-db92-4c79-be80-9d39021ca609 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:53.495254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 577/1169] 10.222.0.26 () {60 vars in 1138 bytes} [Sun Aug 7 00:57:53 2022] POST /identity/v3/domains => generated 267 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:55.582415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b5c61-5849-42fe-a6d8-6708e3881616 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:55.582918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b5c61-5849-42fe-a6d8-6708e3881616 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:55.582918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b5c61-5849-42fe-a6d8-6708e3881616 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:55.584217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 593/1170] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:55.590005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:55.590444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:55.590543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:55.609231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:55.610443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:55.676702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 578/1171] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:57:55.700056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b5c61-5849-42fe-a6d8-6708e3881616 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:55.831267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b5c61-5849-42fe-a6d8-6708e3881616 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:55.868432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d853cb8-8bcb-4989-8a8a-b3f4739cd1e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:55.869494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d853cb8-8bcb-4989-8a8a-b3f4739cd1e2 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:55.869772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d853cb8-8bcb-4989-8a8a-b3f4739cd1e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:55.870047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d853cb8-8bcb-4989-8a8a-b3f4739cd1e2 None admin] PATH_INFO: `/v3/domains/6b35999bda2344a7bb3399a0bbfce371` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:55.880728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d853cb8-8bcb-4989-8a8a-b3f4739cd1e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=6b35999bda2344a7bb3399a0bbfce371)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:55.882162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d853cb8-8bcb-4989-8a8a-b3f4739cd1e2 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:55.883793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 594/1172] 10.222.0.26 () {56 vars in 1183 bytes} [Sun Aug 7 00:57:55 2022] GET /identity/v3/domains/6b35999bda2344a7bb3399a0bbfce371 => generated 267 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:57:55.889722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:55.985545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:55.988603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d89e5b-4baa-4f8b-a09c-ffcbc078943d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:56.028608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe060f1-7317-4705-a7d8-02ecdf96a459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:56.029191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe060f1-7317-4705-a7d8-02ecdf96a459 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:56.029476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe060f1-7317-4705-a7d8-02ecdf96a459 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:56.029753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe060f1-7317-4705-a7d8-02ecdf96a459 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:56.030415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe060f1-7317-4705-a7d8-02ecdf96a459 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:56.031865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe060f1-7317-4705-a7d8-02ecdf96a459 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:56.059979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 579/1173] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:57:55 2022] POST /identity/v3/projects => generated 361 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:57:58.132547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac72a94-41e9-4107-bf6f-fb89a341c081 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:58.133110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac72a94-41e9-4107-bf6f-fb89a341c081 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:58.133110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac72a94-41e9-4107-bf6f-fb89a341c081 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:58.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 595/1174] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:57:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:57:58.139684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:58.140146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:58.140274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:58.157901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:57:58.159076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:57:58.228607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 580/1175] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:57:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:57:58.253187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac72a94-41e9-4107-bf6f-fb89a341c081 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:58.280351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:58.360821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac72a94-41e9-4107-bf6f-fb89a341c081 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:58.396759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c6ce31-2340-41bb-8a07-3dfdb9e5f32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:58.397352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c6ce31-2340-41bb-8a07-3dfdb9e5f32b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:58.397627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c6ce31-2340-41bb-8a07-3dfdb9e5f32b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:58.397898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c6ce31-2340-41bb-8a07-3dfdb9e5f32b None admin] PATH_INFO: `/v3/domains/6b35999bda2344a7bb3399a0bbfce371` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:58.399003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c6ce31-2340-41bb-8a07-3dfdb9e5f32b None admin] RBAC: Authorizing `identity:get_domain(domain_id=6b35999bda2344a7bb3399a0bbfce371)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:58.401210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c6ce31-2340-41bb-8a07-3dfdb9e5f32b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:58.402788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 596/1176] 10.222.0.26 () {56 vars in 1183 bytes} [Sun Aug 7 00:57:58 2022] GET /identity/v3/domains/6b35999bda2344a7bb3399a0bbfce371 => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:57:58.409092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:57:58.488732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:57:58.507851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa71e124-91c0-4aa3-ad6c-4aa2f63f8260 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:57:58.545423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f70e1de-eec7-4fa1-a070-3eb8c8fb4ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:57:58.546029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f70e1de-eec7-4fa1-a070-3eb8c8fb4ba8 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:57:58.546305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f70e1de-eec7-4fa1-a070-3eb8c8fb4ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:57:58.546573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f70e1de-eec7-4fa1-a070-3eb8c8fb4ba8 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:57:58.547216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f70e1de-eec7-4fa1-a070-3eb8c8fb4ba8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:57:58.548672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f70e1de-eec7-4fa1-a070-3eb8c8fb4ba8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:57:58.571125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 581/1177] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:57:58 2022] POST /identity/v3/users => generated 311 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:58:00.629577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5dfd99-f676-4151-be76-b047256f4cd9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:00.630082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5dfd99-f676-4151-be76-b047256f4cd9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:00.630082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5dfd99-f676-4151-be76-b047256f4cd9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:00.631373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 597/1178] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:58:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:58:00.637577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:00.638137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:00.638137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:00.656752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:58:00.657969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:58:00.726894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 582/1179] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:58:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:58:00.750751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5dfd99-f676-4151-be76-b047256f4cd9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:00.843536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:00.853540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5dfd99-f676-4151-be76-b047256f4cd9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:00.888680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:00.889224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:00.889501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:00.889770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:00.900186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:58:00.904865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:58:00.905942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:58:00.906795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 598/1180] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:58:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:58:00.912515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:00.980204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:01.016619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf25f97f-ec0b-4d06-a870-649a7f265f12 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:01.051617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:01.052230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:01.052537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:01.052810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:01.053474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:58:01.053778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:01.055148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:01.063901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 583/1181] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:58:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:01.069978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:01.174835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd21e11-98c2-48b0-a651-652d7fa1bd83 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:01.210976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:01.211595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:01.212426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:01.212710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] PATH_INFO: `/v3/users/288a99305ff9486c9698a7da8c61905b` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:01.227088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] RBAC: Authorizing `identity:get_user(user_id=288a99305ff9486c9698a7da8c61905b)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:01.228716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:01.230489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 599/1182] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:58:01 2022] GET /identity/v3/users/288a99305ff9486c9698a7da8c61905b => generated 311 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:01.236548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:01.338785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe5e83e-ff2e-4c19-ba99-f85579ba4df8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:01.381451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:01.387972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:01.388605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:01.388884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:01.389155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:01.455950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:01.457346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:01.553178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 584/1183] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:58:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:58:01.559408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:01.571296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:01.657221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6652303a-bffa-4596-8490-d7cfc67d36a6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:01.691999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ca033a-0d76-493c-8793-285b3ef06c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:01.692669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ca033a-0d76-493c-8793-285b3ef06c16 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:01.692950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ca033a-0d76-493c-8793-285b3ef06c16 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:01.693226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ca033a-0d76-493c-8793-285b3ef06c16 None admin] PATH_INFO: `/v3/projects/e685cb11f945431fb4805833caeb4c47` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:01.694351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ca033a-0d76-493c-8793-285b3ef06c16 None admin] RBAC: Authorizing `identity:get_project(project_id=e685cb11f945431fb4805833caeb4c47)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:01.695786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ca033a-0d76-493c-8793-285b3ef06c16 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:01.698119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 600/1184] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:58:01 2022] GET /identity/v3/projects/e685cb11f945431fb4805833caeb4c47 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:01.704705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:01.734647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:01.805034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b3697-168b-4624-bc43-2f0ea3b7bf1b None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:01.839787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:01.840421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:01.840706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:01.840978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:01.841698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=e685cb11f945431fb4805833caeb4c47, user.id=288a99305ff9486c9698a7da8c61905b) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:58:01.842007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:01.843433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da06e3d-c0f6-4585-bf5d-73a9389c21c4 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:01.848785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 585/1185] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:58:01 2022] GET /identity/v3/role_assignments?user.id=288a99305ff9486c9698a7da8c61905b&scope.project.id=e685cb11f945431fb4805833caeb4c47&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:03.850790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e278c45a-f1e3-4d66-949d-8db5cea81cba None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:03.851434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e278c45a-f1e3-4d66-949d-8db5cea81cba None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:03.851434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e278c45a-f1e3-4d66-949d-8db5cea81cba None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:03.852741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 601/1186] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:58:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:58:03.864917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:03.865385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:03.865501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:03.883355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:58:03.884594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:58:03.950917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 586/1187] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:58:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:58:03.974121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e278c45a-f1e3-4d66-949d-8db5cea81cba None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:04.059528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:04.082517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e278c45a-f1e3-4d66-949d-8db5cea81cba None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:04.115652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:04.116193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:04.116501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:04.116775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:04.126822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:58:04.131443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:58:04.132658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:58:04.134435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 602/1188] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:58:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:58:04.139455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:04.165835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:04.251089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d468a73e-99eb-4a9c-b59a-e240095b446f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:04.286223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:04.286857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:04.287126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:04.287420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:04.288146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:58:04.288568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:04.290570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:04.299948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 587/1189] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:58:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:58:04.305765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:04.406688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb81cde-c1ad-49e2-80b2-ebeacc94e7c8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:04.452914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:04.453543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:04.453827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:04.454103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] PATH_INFO: `/v3/users/288a99305ff9486c9698a7da8c61905b` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:04.455228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] RBAC: Authorizing `identity:get_user(user_id=288a99305ff9486c9698a7da8c61905b)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:04.456694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:04.458484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 603/1190] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:58:04 2022] GET /identity/v3/users/288a99305ff9486c9698a7da8c61905b => generated 311 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:04.465335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:04.551557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:04.567708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2825987-1f7f-438a-8cca-9971e36988bc None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:04.601167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555fe0df-cab4-4100-8604-ef085430decf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:04.601795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555fe0df-cab4-4100-8604-ef085430decf None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:04.602078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555fe0df-cab4-4100-8604-ef085430decf None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:04.602384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555fe0df-cab4-4100-8604-ef085430decf None admin] PATH_INFO: `/v3/projects/e685cb11f945431fb4805833caeb4c47` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:04.603491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555fe0df-cab4-4100-8604-ef085430decf None admin] RBAC: Authorizing `identity:get_project(project_id=e685cb11f945431fb4805833caeb4c47)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:04.604959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555fe0df-cab4-4100-8604-ef085430decf None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:04.606604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 588/1191] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:58:04 2022] GET /identity/v3/projects/e685cb11f945431fb4805833caeb4c47 => generated 361 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:04.612840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:04.622354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:04.730134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbfee04-4c66-47af-b2d9-36c45254c1bb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:04.764705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654bb587-01a4-4a7c-97f4-a6039b7a1f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:04.765354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654bb587-01a4-4a7c-97f4-a6039b7a1f2b None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:04.765629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654bb587-01a4-4a7c-97f4-a6039b7a1f2b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:04.765899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654bb587-01a4-4a7c-97f4-a6039b7a1f2b None admin] PATH_INFO: `/v3/projects/e685cb11f945431fb4805833caeb4c47/users/288a99305ff9486c9698a7da8c61905b/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:04.767807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654bb587-01a4-4a7c-97f4-a6039b7a1f2b None admin] RBAC: Authorizing `identity:create_grant(project_id=e685cb11f945431fb4805833caeb4c47, user_id=288a99305ff9486c9698a7da8c61905b, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:04.769312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654bb587-01a4-4a7c-97f4-a6039b7a1f2b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:04.782279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 604/1192] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:58:04 2022] PUT /identity/v3/projects/e685cb11f945431fb4805833caeb4c47/users/288a99305ff9486c9698a7da8c61905b/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:58:06.806684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e041a-48d2-4ca8-be9f-469314eff141 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:06.807247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e041a-48d2-4ca8-be9f-469314eff141 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:06.807247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e041a-48d2-4ca8-be9f-469314eff141 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:06.808609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 589/1193] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:58:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:58:06.814411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:06.814924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:06.815026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:06.833350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:58:06.834564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:58:06.903073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:06.913412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 605/1194] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:58:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:58:06.937593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e041a-48d2-4ca8-be9f-469314eff141 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:07.058182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e041a-48d2-4ca8-be9f-469314eff141 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:07.092489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:07.093123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:07.093400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:07.093672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:07.104763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:58:07.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:58:07.110439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:58:07.111332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 590/1195] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:58:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:58:07.117545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:07.215789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab1f447-08ed-44a0-af02-e437454f8307 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:07.249404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:07.250017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:07.250297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:07.250568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:07.251257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:58:07.251558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:07.252959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:07.261689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 606/1196] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:58:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:07.267641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:07.284714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:07.372597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516afbe1-ce95-4faf-b871-89a9d23768f6 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:07.406882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:07.407508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:07.407785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:07.408057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] PATH_INFO: `/v3/users/288a99305ff9486c9698a7da8c61905b` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:07.409310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] RBAC: Authorizing `identity:get_user(user_id=288a99305ff9486c9698a7da8c61905b)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:07.410732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:07.412361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 591/1197] 10.222.0.26 () {56 vars in 1177 bytes} [Sun Aug 7 00:58:07 2022] GET /identity/v3/users/288a99305ff9486c9698a7da8c61905b => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:07.418497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:07.482300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:07.524987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee72014-4d77-41f6-9a97-805508e7f8c7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:07.560979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:07.561587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:07.561864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:07.562138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:07.636416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:07.637576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:07.694166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:07.743241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 607/1198] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:58:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:58:07.749756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:07.838261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:07.859433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85842fdc-05ae-4466-9383-155b03375f7d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:07.894486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d120261-57fb-4d7d-8314-854e4cb67815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:07.895039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d120261-57fb-4d7d-8314-854e4cb67815 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:07.895324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d120261-57fb-4d7d-8314-854e4cb67815 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:07.895602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d120261-57fb-4d7d-8314-854e4cb67815 None admin] PATH_INFO: `/v3/projects/e685cb11f945431fb4805833caeb4c47` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:07.896773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d120261-57fb-4d7d-8314-854e4cb67815 None admin] RBAC: Authorizing `identity:get_project(project_id=e685cb11f945431fb4805833caeb4c47)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:07.898220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d120261-57fb-4d7d-8314-854e4cb67815 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:07.900522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 592/1199] 10.222.0.26 () {56 vars in 1186 bytes} [Sun Aug 7 00:58:07 2022] GET /identity/v3/projects/e685cb11f945431fb4805833caeb4c47 => generated 361 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:07.906942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:58:08.016665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e68073-b691-4c49-ba78-95ce8ea06964 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:58:08.052782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:58:08.053398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:58:08.053680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:58:08.053952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:58:08.054690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=e685cb11f945431fb4805833caeb4c47, user.id=288a99305ff9486c9698a7da8c61905b) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:58:08.054998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:58:08.056575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f261b96f-7264-4b81-9b48-49e5e504c40b None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:58:08.065326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 608/1200] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:58:07 2022] GET /identity/v3/role_assignments?user.id=288a99305ff9486c9698a7da8c61905b&scope.project.id=e685cb11f945431fb4805833caeb4c47&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:58:17.890514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=25 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:58:18.063145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:08.695699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837b8439-687e-4d76-be60-109c77a5e6ee None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:08.696732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837b8439-687e-4d76-be60-109c77a5e6ee None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:08.696732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837b8439-687e-4d76-be60-109c77a5e6ee None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:08.698641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 593/1201] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:08.705363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:08.706251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:08.706251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:08.736402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:08.738919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:08.889769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 609/1202] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:59:08.928295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837b8439-687e-4d76-be60-109c77a5e6ee None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:09.130895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837b8439-687e-4d76-be60-109c77a5e6ee None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:09.222123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:09.223168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:09.223635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:09.224280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:09.250430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:09.253532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:09.283148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:09.285953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:09.287777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:09.287777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:09.287777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:09.292855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 594/1203] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 365 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:09.300077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:09.500804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0206759d-663c-4d11-aa5c-3026b803d803 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:09.548298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:09.563005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:09.563005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:09.563005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:09.563005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:09.563767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:09.563767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:09.564521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c4c0a-e868-4fec-9a8a-c9c50fbaf394 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:09.592419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 610/1204] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:09.597493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:09.707601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:09.837794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c88dd36-f46b-4605-beb7-a6a5a9b3a00a None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:09.903546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57189-77c1-44a3-9578-081e6c2ec89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:09.904789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57189-77c1-44a3-9578-081e6c2ec89a None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:09.905229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57189-77c1-44a3-9578-081e6c2ec89a None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:09.905763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57189-77c1-44a3-9578-081e6c2ec89a None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:09.906644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57189-77c1-44a3-9578-081e6c2ec89a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:09.908864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57189-77c1-44a3-9578-081e6c2ec89a None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:09.947202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 595/1205] 10.222.0.26 () {60 vars in 1132 bytes} [Sun Aug 7 00:59:09 2022] POST /identity/v3/users => generated 251 bytes in 350 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:59:12.842151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d58ec-859b-4674-a42e-bb97299a22c0 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:12.843979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d58ec-859b-4674-a42e-bb97299a22c0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:12.843979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d58ec-859b-4674-a42e-bb97299a22c0 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:12.845214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 611/1206] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:59:12.853534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:12.854776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:12.854776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:12.876031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:12.877467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:13.000055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 596/1207] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:13.048838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d58ec-859b-4674-a42e-bb97299a22c0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:13.244506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d58ec-859b-4674-a42e-bb97299a22c0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:13.330213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:13.331501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:13.332075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:13.332638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:13.359270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:59:13.371016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:13.373321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:13.374792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 612/1208] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:59:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:13.385636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:13.478206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:13.572812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a61b1c-4cc4-4d0d-bdc0-77bee46a6011 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:13.611648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637259a8-b519-4842-a447-0da627185d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:13.612313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637259a8-b519-4842-a447-0da627185d08 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:13.612633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637259a8-b519-4842-a447-0da627185d08 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:13.612917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637259a8-b519-4842-a447-0da627185d08 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:13.613610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637259a8-b519-4842-a447-0da627185d08 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:13.613906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637259a8-b519-4842-a447-0da627185d08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:13.615359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637259a8-b519-4842-a447-0da627185d08 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:13.636507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 597/1209] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:59:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:13.638998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:13.752499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:13.887826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7652378-4581-4abb-9409-f102589277c8 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:13.944170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:13.945274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:13.945776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:13.946778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:13.967474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:13.970043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:13.993563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:13.995491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:13.998063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:13.998063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:13.998063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:13.998063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 613/1210] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 358 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:59:14.004046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637259a8-b519-4842-a447-0da627185d08 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:14.189951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637259a8-b519-4842-a447-0da627185d08 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:14.192985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:14.267120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:14.268540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:14.269210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:14.269838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:14.271553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:14.272343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:14.275195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:14.310417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 598/1211] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:14.313377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:14.517182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa854965-86d2-42c7-98e4-89e755bddff3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:14.519914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:14.588491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:14.589645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:14.590161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:14.590654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] PATH_INFO: `/v3/users/placement` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:14.608755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:14.612049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:59:14.632491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:14.635181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:14.637306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 614/1212] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:14.643244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:14.788625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d46b17-0415-4376-a0c8-27dc8e09adf2 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:14.833275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:14.833905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:14.834203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:14.834523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:14.835193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:14.835495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:14.837635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:14.860078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 599/1213] 10.222.0.26 () {56 vars in 1143 bytes} [Sun Aug 7 00:59:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:14.870254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:15.062799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecee070c-b83f-4560-b3d7-db87c3aa8e5f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:15.130798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:15.131739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:15.132184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:15.132619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:15.188447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:15.244775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:15.247592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:15.447589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 615/1214] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:59:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 578 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:59:15.457742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:15.497653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:15.676184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3db8757-7ae8-4625-9211-369ca2a80ae8 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:15.754825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:15.756043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:15.756735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:15.757308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:15.780952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:15.785112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:15.816556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:15.818593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:15.820725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:15.820725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:15.820725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:15.820725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 600/1215] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:15.827491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:15.956589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:16.020489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73665751-6d5e-4bc9-a241-42877494f810 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:16.066083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:16.067577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:16.068080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:16.068612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:16.069551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:16.070128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:16.072449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:16.087424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 616/1216] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:16.099186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:16.247936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:16.319666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32fb74d-9e88-45ad-9e10-790316859674 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:16.385262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:16.386603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:16.386603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:16.386603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:16.395015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:16.397902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:59:16.421620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:16.422780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:16.423782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:16.432699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 601/1217] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:59:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:16.434927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:16.639933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a3a3d-a3d8-46de-b2df-13e8dfe7af11 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:16.683288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:16.706580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:16.707583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:16.708058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:16.708572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:16.709892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:16.710600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:16.713772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73001525-a33e-4558-8934-ca488e756c4a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:16.729364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 617/1218] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:59:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:16.736662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:16.925417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e28c98-0343-462b-80be-132bbb81491f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:16.989963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:17.000005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:17.000851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:17.001116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:17.001412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:17.002347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=06551751330a406f8f0003153245b6a9) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:17.002714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:17.005290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8fe41a-8630-4d16-833e-4d8a917fffb6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:17.034106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 602/1219] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:59:16 2022] GET /identity/v3/role_assignments?user.id=06551751330a406f8f0003153245b6a9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 259 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:19.799198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6f5a7f-f1e0-4800-bed7-a5d37439f3d9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:19.800096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6f5a7f-f1e0-4800-bed7-a5d37439f3d9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:19.800096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6f5a7f-f1e0-4800-bed7-a5d37439f3d9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:19.802416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 618/1220] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:19.810915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:19.812050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:19.812050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:19.839796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:19.841292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:19.940481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 603/1221] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:59:19.959472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6f5a7f-f1e0-4800-bed7-a5d37439f3d9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:20.099846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6f5a7f-f1e0-4800-bed7-a5d37439f3d9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:20.150182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:20.150855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:20.151145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:20.151510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:20.167053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:59:20.175701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:20.177528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:20.178965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 619/1222] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:59:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:20.188203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:20.350723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86754a06-052b-4217-ba08-69b80aee4a1f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:20.387923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:20.407681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:20.408755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:20.409280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:20.409770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:20.410997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] RBAC: Adding query filter params (name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:20.411543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:20.414150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:20.429111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 604/1223] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:59:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:20.442494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:20.514302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:20.605870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba2c48c-d609-42ef-9e98-27e4a2b9a5f6 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:20.666589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:20.667403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:20.667913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:20.668395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:20.682513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:20.684169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:20.702877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:20.705600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:20.707793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:20.707793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:20.707793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:20.707793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 620/1224] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:20.713381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:20.912648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa69039-9000-4e10-8e57-e9010b3aff13 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:20.984380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:20.985580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:20.986203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:20.986600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:20.987803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:20.988301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:20.990697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:21.012752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 605/1225] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:21.023067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:21.210564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:21.225084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9256baa7-fab7-43b6-80ed-9f213dcb7e17 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:21.306319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:21.307993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:21.308548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:21.309110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:21.326154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:21.328887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:59:21.352908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:21.354934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:21.357349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 621/1226] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:21.364668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:21.455081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:21.561220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031ccca-fd6a-40c4-8ee4-fb2a0d334e79 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:21.627150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:21.628150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:21.628651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:21.629076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:21.630324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:21.630809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:21.634042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:21.664876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 606/1227] 10.222.0.26 () {56 vars in 1143 bytes} [Sun Aug 7 00:59:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:21.672606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:21.874532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bf4cc9-6355-4238-b781-9c87dacdc086 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:21.960008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:21.961083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:21.961517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:21.962466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:21.985388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:21.988520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:22.016206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:22.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:22.019537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:22.019537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:22.019537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:22.019537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 622/1228] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 347 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:22.029477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:22.239460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98530213-1ccb-4a9e-b707-cb0aea74cd81 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:22.245334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:22.313915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:22.314857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:22.315243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:22.315615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:22.316480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:22.316912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:22.318548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:22.334150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 607/1229] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:22.340974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:22.353596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:22.549183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a53a0c0-f84e-4522-a86f-73cc63051bc7 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:22.617232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:22.618930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:22.619465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:22.620002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:22.635574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:22.638405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:59:22.659918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:22.661524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:22.663264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 623/1230] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:59:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:59:22.673258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:22.891000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479039e8-3e9d-4a68-a645-bf8dd9aa1069 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:22.970922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:22.971958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:22.972521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:22.973013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:22.974255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:22.974809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:22.977607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf3e55e-161c-44b1-aaeb-40282ee9d54e None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:23.007105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 608/1231] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:59:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:23.009456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:23.155155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:23.233875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c837cc4-b292-4ae0-92f8-1f64b1bf3c55 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:23.301944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec96479a-5650-4f03-aaa3-7c1de4541922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:23.303080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec96479a-5650-4f03-aaa3-7c1de4541922 None admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:23.303554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec96479a-5650-4f03-aaa3-7c1de4541922 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:23.304048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec96479a-5650-4f03-aaa3-7c1de4541922 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/06551751330a406f8f0003153245b6a9/roles/2ab58373774e4258b896999b6dfd49df` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:23.331925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec96479a-5650-4f03-aaa3-7c1de4541922 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=06551751330a406f8f0003153245b6a9, role_id=2ab58373774e4258b896999b6dfd49df)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:23.334720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec96479a-5650-4f03-aaa3-7c1de4541922 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:23.359760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 624/1232] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:59:23 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/06551751330a406f8f0003153245b6a9/roles/2ab58373774e4258b896999b6dfd49df => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:59:27.037069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab02491-d48c-402c-94d4-4ee9c2d19f73 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:27.037967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab02491-d48c-402c-94d4-4ee9c2d19f73 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:27.037967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab02491-d48c-402c-94d4-4ee9c2d19f73 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:27.044316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 609/1233] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:27.050448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:27.051327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:27.051327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:27.085247 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:27.086938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:27.223782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 625/1234] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 00:59:27.266644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab02491-d48c-402c-94d4-4ee9c2d19f73 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:27.340429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:27.490937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab02491-d48c-402c-94d4-4ee9c2d19f73 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:27.555111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:27.556731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:27.557188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:27.557664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:27.578121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:59:27.588169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:27.590248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:27.591310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 610/1235] 10.222.0.26 () {56 vars in 1102 bytes} [Sun Aug 7 00:59:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:27.599496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:27.686398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:27.801977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e01ebb-f922-4469-a039-b1fb853dbdc9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:27.876135 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:27.877121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:27.877530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:27.878009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:27.879384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] RBAC: Adding query filter params (name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:27.880019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:27.882757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:27.907162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 626/1236] 10.222.0.26 () {56 vars in 1103 bytes} [Sun Aug 7 00:59:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:27.909108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:28.111540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:28.160283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56be9d81-90b2-45ee-9294-0138413ca5c5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:28.244571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:28.245616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:28.246004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:28.246451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:28.269041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:28.271679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:28.308263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:28.310147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:28.312220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:28.312220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:28.312220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:28.320024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 611/1237] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 412 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:28.322204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:28.526515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a84d28-4ff5-4116-bc85-d97129186f78 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:28.557930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:28.616625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:28.617478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:28.617745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:28.618024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:28.619207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:28.619506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:28.620979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:28.653422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 627/1238] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:28.667668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:28.883110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ce4a0f-873d-49d3-bdf4-929ee9c85948 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:28.952517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:28.953634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:28.954034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:28.954473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:28.959789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:28.974257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:28.977190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:59:29.000706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:29.002828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:29.005143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 612/1239] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:29.018198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:29.240584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a807a1d-613f-4f2e-9611-fe5f8c2439a3 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:29.308504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:29.309655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:29.310234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:29.310691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:29.312158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:29.312764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:29.315549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:29.360411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 628/1240] 10.222.0.26 () {56 vars in 1143 bytes} [Sun Aug 7 00:59:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:29.386464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:29.618833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ce175f-02d2-41a7-bc0e-287a27aab1b9 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:29.688471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:29.689475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:29.690021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:29.690468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:29.844048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:29.862731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:29.865651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:30.077456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 613/1241] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:59:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 692 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:59:30.085162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:30.140070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:30.295164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c194983b-8113-4c40-a066-50a3e3635ddd None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:30.364167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:30.365431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:30.365685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:30.366163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:30.391443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:30.394527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:30.425886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:30.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:30.429615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:30.429615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:30.429615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:30.429615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 629/1242] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:30.437908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:30.629765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:30.641759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1785711d-990d-4392-ac38-53a75e111137 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:30.715051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:30.716102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:30.716638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:30.717137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:30.718349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:30.718900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:30.721458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:30.752115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 614/1243] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:30.764538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:30.935402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:30.985697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f287a7-e58b-4b95-88bf-9c2971e5dad1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:31.054273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:31.056426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:31.056710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:31.057287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:31.072976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:31.075944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:59:31.099646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:31.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:31.107127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 630/1244] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:59:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:31.115981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:31.402118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977b0cd3-7862-44c8-9b6e-7de2903908fa None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:31.479821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:31.481268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:31.483401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:31.483934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:31.485322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:31.485859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:31.488832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb74148-59a8-4888-a6dd-9c7d69c7c5de None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:31.516077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 615/1245] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:59:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 404 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:31.528281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:31.725997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09dd494-bfa7-402f-9b1d-2606218da3f9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:31.798060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:31.799087 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:31.799607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:31.800024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:31.801404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] RBAC: Adding query filter params (role.id=2ab58373774e4258b896999b6dfd49df, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=06551751330a406f8f0003153245b6a9) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:31.801875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:31.804574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb27c51-e670-461b-aa6c-511513bd0e73 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:31.816693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 631/1246] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:59:31 2022] GET /identity/v3/role_assignments?user.id=06551751330a406f8f0003153245b6a9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=2ab58373774e4258b896999b6dfd49df => generated 608 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:34.738069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8eb1bc8-a626-4114-8e59-ccc212b5c751 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:34.738924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8eb1bc8-a626-4114-8e59-ccc212b5c751 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:34.738924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8eb1bc8-a626-4114-8e59-ccc212b5c751 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:34.742086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 616/1247] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:34.747471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:34.747999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:34.748500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:34.759917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:34.782003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:34.784213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:34.912265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 632/1248] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:34.939293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8eb1bc8-a626-4114-8e59-ccc212b5c751 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:34.983454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:35.164464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8eb1bc8-a626-4114-8e59-ccc212b5c751 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:35.252605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:35.253289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:35.253602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:35.253887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:35.283509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:59:35.291431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:35.293947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:35.295604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 617/1249] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:59:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:35.302970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:35.495672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8cd03e-fa8b-49e2-9a50-d8e7cf9d41b9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:35.557516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:35.568061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:35.569419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:35.570447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:35.570447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:35.571596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:35.572374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:35.573907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:35.586003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 633/1250] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:59:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:35.598926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:35.768509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:35.823207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca49220e-d71c-4c09-83e4-d3b846e411ee None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:35.896932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:35.898189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:35.898686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:35.899705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:35.923192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:35.925229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:35.951029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:35.952396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:35.954181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:35.954181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:35.954181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:35.954181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 618/1251] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 355 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:35.960409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:36.129465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3d196e-c8fe-49b8-9b84-0ca80b0ba056 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:36.203073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:36.204720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:36.205430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:36.206103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:36.207439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:36.208209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:36.210926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:36.237375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 634/1252] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:36.249257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:36.493638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfe4d83-6186-4a8a-86ea-f612615aab57 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:36.566345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:36.581211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:36.582748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:36.583431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:36.584111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:36.601224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:36.605079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:59:36.628663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:36.630017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:36.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 619/1253] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:36 2022] GET /identity/v3/users/placement => generated 87 bytes in 384 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:59:36.642796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:36.744797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:36.836391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce6670a-c078-4f6d-b8bd-70fb6b9eaa22 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:36.896251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:36.897067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:36.897350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:36.897630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:36.898451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:36.898758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:36.901158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:36.930104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 635/1254] 10.222.0.26 () {56 vars in 1143 bytes} [Sun Aug 7 00:59:36 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:36.940092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:37.132407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dec7186-f7dc-4973-a226-d9a7860a54f4 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:37.207164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:37.208053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:37.208575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:37.209008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:37.343919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:37.370407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:37.372399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:37.556075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 620/1255] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:59:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 617 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 07 00:59:37.563653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:37.771710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601565a7-1235-445d-8087-77a8573e2f6f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:37.790668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:37.834978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:37.835926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:37.836635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:37.837682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:37.854955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:37.856895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:37.885182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:37.887492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:37.889990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:37.889990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:37.889990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:37.889990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 636/1256] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:59:37.896908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:38.065181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:38.089794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b5a800-351b-48ee-bc14-7b3e6f86f4f3 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:38.159848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:38.161191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:38.161830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:38.162402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:38.163559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:38.164098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:38.167239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:38.201717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 621/1257] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:38.208679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:38.400520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b698f3d4-9d7c-4894-bb96-bd35dec8f6e1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:38.447306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:38.448038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:38.448416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:38.448690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:38.459316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:38.459593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:59:38.473766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:38.475640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:38.477866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 637/1258] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:59:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:38.486729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:38.681943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cfeb05-7c56-4ba5-81cd-ab93d2620818 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:38.749598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:38.750655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:38.751207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:38.751653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:38.753050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:38.753615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:38.756505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed01fa9a-cf74-4ae4-951a-a3f48d5f0634 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:38.784182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 622/1259] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:59:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:38.796128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:38.855341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:38.987642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb9649c-473a-41f1-80c1-cbe1a503835d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:39.047231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:39.048343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:39.048839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:39.049327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:39.050833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=06551751330a406f8f0003153245b6a9) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:39.051417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:39.054284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be18ffb9-0d98-46e1-8a47-ddfd9139791a None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:39.065341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 638/1260] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:59:38 2022] GET /identity/v3/role_assignments?user.id=06551751330a406f8f0003153245b6a9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=b0ec0093717c472da3640ec7121208d4 => generated 259 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:42.672121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e7eaf9-452d-48c8-b942-0f06179de467 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:42.673012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e7eaf9-452d-48c8-b942-0f06179de467 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:42.673012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e7eaf9-452d-48c8-b942-0f06179de467 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:42.675518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 623/1261] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:42.686541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:42.687642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:42.687839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:42.719968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:42.722063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:42.848097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 639/1262] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:42.884350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e7eaf9-452d-48c8-b942-0f06179de467 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:42.904399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:43.025744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e7eaf9-452d-48c8-b942-0f06179de467 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:43.067685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:43.068527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:43.068901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:43.069288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:43.082826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:59:43.088047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:43.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:43.091714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 624/1263] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:59:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:43.099423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:43.121348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:43.260266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c570da-331e-4a3e-aa98-80254a9cfaeb None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:43.302826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:43.303533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:43.303940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:43.304369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:43.305179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:43.305624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:43.307237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:43.317477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 640/1264] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:59:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:43.327583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:43.417349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:43.456996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc58d1b6-d5d0-4b29-90c4-90c5b9223fc1 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:43.498521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:43.499295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:43.499638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:43.499987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:43.514070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:43.515720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:43.538121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:43.540518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:43.542229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:43.542229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:43.542229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:43.542229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 625/1265] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:43.548652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:43.709568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:43.747448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b82e78-6f82-47e1-a1f5-1e588f24ef25 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:43.814853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:43.815802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:43.816319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:43.816792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:43.818018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:43.818559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:43.821250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:43.845815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 641/1266] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:43.855437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:44.074785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49573396-6e3d-4e8b-86c8-463fdb93165d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:44.118362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:44.144896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:44.146342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:44.147084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:44.147561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:44.164066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:44.167419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:59:44.189760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:44.191641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:44.193395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 626/1267] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:43 2022] GET /identity/v3/users/placement => generated 87 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:44.201400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:44.377405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e241e10-cea4-4b88-8a5d-9f76be1947b1 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:44.448751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:44.449850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:44.450347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:44.450843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:44.452294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:44.452837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:44.455645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:44.473993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:44.490408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 642/1268] 10.222.0.26 () {56 vars in 1143 bytes} [Sun Aug 7 00:59:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:44.498072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:44.678287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf9cf5b-16eb-44ca-8bf6-005f2f671b05 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:44.726557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:44.727286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:44.728132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:44.728495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:44.743609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:44.746007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:44.764460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:44.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:44.768503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:44.768503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:44.768503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:44.768503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 627/1269] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:44.772388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:44.975994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e761d78a-c115-46aa-b86b-658251e50d15 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:45.043816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:45.044850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:45.045320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:45.045774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:45.046928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:45.047463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:45.050041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:45.072051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 643/1270] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:45.081098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:45.128587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:45.258385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc6d8b-19d9-4feb-83b3-be8a4e083107 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:45.323536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:45.324688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:45.325199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:45.325710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:45.338981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:45.341758 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:59:45.363921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:45.365933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:45.367940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 628/1271] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:59:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 07 00:59:45.376421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:45.492079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:45.587672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb52dc2-395c-418c-ace6-2d316628e7bf None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:45.655310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:45.656283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:45.656694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:45.657115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:45.658325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:45.658823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:45.661424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e77d238-3f7a-4cb9-a356-f86e327fb2b7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:45.685219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 644/1272] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:59:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:45.696838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:45.871331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:45.890278 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af19cc43-b75e-41c5-ace1-e4b474a2d796 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:45.958565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c03a8c-9474-42da-9b49-3647fecf4e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:45.959759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c03a8c-9474-42da-9b49-3647fecf4e00 None admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:45.960301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c03a8c-9474-42da-9b49-3647fecf4e00 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:45.960794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c03a8c-9474-42da-9b49-3647fecf4e00 None admin] PATH_INFO: `/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/06551751330a406f8f0003153245b6a9/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:45.965009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c03a8c-9474-42da-9b49-3647fecf4e00 None admin] RBAC: Authorizing `identity:create_grant(project_id=2f39278bb4be4af68b716f8d549c5155, user_id=06551751330a406f8f0003153245b6a9, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:45.967791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c03a8c-9474-42da-9b49-3647fecf4e00 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:45.989017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 629/1273] 10.222.0.26 () {58 vars in 1439 bytes} [Sun Aug 7 00:59:45 2022] PUT /identity/v3/projects/2f39278bb4be4af68b716f8d549c5155/users/06551751330a406f8f0003153245b6a9/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 00:59:48.948809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e9365-83f5-4891-9189-0548eee7fd3b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:48.949379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e9365-83f5-4891-9189-0548eee7fd3b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:48.949609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e9365-83f5-4891-9189-0548eee7fd3b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:48.951829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 645/1274] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:48.958867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:48.959469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:48.959745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:48.989305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:48.991323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:49.067012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 630/1275] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:49.091483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e9365-83f5-4891-9189-0548eee7fd3b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:49.227897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e9365-83f5-4891-9189-0548eee7fd3b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:49.249491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:49.267840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:49.268504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:49.268788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:49.269064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:49.279999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 07 00:59:49.284757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:49.286572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:49.288189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 646/1276] 10.222.0.26 () {56 vars in 1096 bytes} [Sun Aug 7 00:59:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:49.292329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:49.377722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:49.398208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95900541-fe2e-4b15-af27-ed72aad24b2f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:49.434381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:49.435021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:49.435302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:49.435578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:49.436337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:49.436648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:49.438324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:49.448145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 631/1277] 10.222.0.26 () {56 vars in 1099 bytes} [Sun Aug 7 00:59:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:49.454477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:49.568472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e37911-1f97-421e-b80f-f4a9f76644b9 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:49.607440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:49.608083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:49.608395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:49.608674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:49.620553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:49.622054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:49.638831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:49.640688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:49.642383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:49.642383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:49.642383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:49.642383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 647/1278] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:49.646117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:49.776493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b093c14c-d6f5-4948-b601-d00b15178307 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:49.818313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:49.818995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:49.819205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:49.819479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:49.820126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:49.820455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:49.821871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:49.838746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 632/1279] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:49.844904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:49.886805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:49.965724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c62ac8-3706-4401-b013-91fc211174bb None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:50.006642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:50.007469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:50.007773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:50.008176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:50.021424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:50.024109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 07 00:59:50.043326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:50.044613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:50.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 648/1280] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:50.053161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:50.060543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:50.169499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783ed9f9-c710-42d9-ad95-048892f68e98 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:50.208909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:50.209536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:50.209820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:50.210093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:50.210843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:50.211147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:50.212684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:50.232855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 633/1281] 10.222.0.26 () {56 vars in 1143 bytes} [Sun Aug 7 00:59:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:50.238804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:50.363272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:50.379647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae49d2-b84c-4897-8bbd-f196ab311b22 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:50.422869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:50.423397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:50.423675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:50.423948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:50.502234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:50.503715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:50.613181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:50.630792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 649/1282] 10.222.0.26 () {58 vars in 1282 bytes} [Sun Aug 7 00:59:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 07 00:59:50.639187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:50.735043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:50.785828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f546845d-054b-46ea-8ad1-69d242ab33d5 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:50.824516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:50.825252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:50.825528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:50.825840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:50.837287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:50.838842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 07 00:59:50.856136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:50.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 07 00:59:50.859072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 07 00:59:50.859072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 07 00:59:50.859072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:50.859072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 634/1283] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 00:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:50.863886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:50.973128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd1b3f0-846d-479b-98c6-d483514d972f None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:51.012712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:51.013265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:51.013609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:51.013826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:51.014558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:51.014901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:51.016491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:51.032175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 650/1284] 10.222.0.26 () {56 vars in 1109 bytes} [Sun Aug 7 00:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:51.038712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:51.168134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4aca970-50b9-49dd-b9c4-740671027f4d None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:51.185147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:51.207058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:51.207702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:51.207985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:51.208291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:51.215770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:51.217266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 07 00:59:51.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:51.231814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: ERROR keystone.server.flask.application Aug 07 00:59:51.233637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 635/1285] 10.222.0.26 () {56 vars in 1111 bytes} [Sun Aug 7 00:59:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:51.237283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:51.294418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:51.352034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c9c35-2861-47dc-9269-d3afb591e344 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:51.390229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:51.390910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:51.391240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:51.391573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:51.392376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:51.392724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:51.394216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74368604-abb8-4997-9888-0ef898d502f7 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:51.408167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 651/1286] 10.222.0.26 () {56 vars in 1148 bytes} [Sun Aug 7 00:59:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:51.415258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:51.555252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b91680-d844-44b7-8d34-a80ce05f0849 None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:51.600725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:51.601727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:51.602084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:51.602616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:51.603810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] RBAC: Adding query filter params (role.id=b0ec0093717c472da3640ec7121208d4, scope.project.id=2f39278bb4be4af68b716f8d549c5155, user.id=06551751330a406f8f0003153245b6a9) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:51.604311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:51.606745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2013d7-cba3-40b3-ba4f-a7c85b1af8ef None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:51.616405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 636/1287] 10.222.0.26 () {56 vars in 1374 bytes} [Sun Aug 7 00:59:51 2022] GET /identity/v3/role_assignments?user.id=06551751330a406f8f0003153245b6a9&scope.project.id=2f39278bb4be4af68b716f8d549c5155&role.id=b0ec0093717c472da3640ec7121208d4 => generated 608 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:53.754085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd84bf-f524-4889-b123-0d02ecfe7b5f None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:53.754661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd84bf-f524-4889-b123-0d02ecfe7b5f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:53.754661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cd84bf-f524-4889-b123-0d02ecfe7b5f None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:53.755899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 652/1288] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:59:53.762984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:53.763671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:53.763925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:53.786219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:53.787526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:53.823879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:53.890369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 637/1289] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:53.913942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd84bf-f524-4889-b123-0d02ecfe7b5f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:54.072633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cd84bf-f524-4889-b123-0d02ecfe7b5f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:54.091260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:54.123351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:54.124014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:54.124327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:54.124607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:54.125209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:54.126618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:54.138094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:54.140327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:54.142403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 653/1290] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:59:53 2022] GET /identity/v3/services/placement => generated 90 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:54.148191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:54.286231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371d99f-8e90-4343-8e31-ddfd2992985b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:54.340982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:54.341618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:54.341905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:54.342179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:54.342850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:54.343156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:54.344666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f865ec-95de-4933-8e2c-4a42a3a9ea58 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:54.350716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 638/1291] 10.222.0.26 () {56 vars in 1116 bytes} [Sun Aug 7 00:59:54 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:54.357825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:54.473551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941817f9-8cb9-4f11-a58a-58e6d4d5ec65 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:54.522443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:54.523067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:54.523343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:54.523612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] PATH_INFO: `/v3/services` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:54.524502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:54.524803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:54.526547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c18a439-398e-44bd-b8ee-57089631f1c3 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:54.532642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 654/1292] 10.222.0.26 () {56 vars in 1116 bytes} [Sun Aug 7 00:59:54 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 00:59:56.593814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8179583-db95-4446-ae55-37c59ae09e17 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:56.594596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8179583-db95-4446-ae55-37c59ae09e17 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:56.594596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8179583-db95-4446-ae55-37c59ae09e17 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:56.595605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 639/1293] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 00:59:56.604015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd8d532-aebb-4b82-8f39-613411fff2b3 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:56.604968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd8d532-aebb-4b82-8f39-613411fff2b3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:56.604968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd8d532-aebb-4b82-8f39-613411fff2b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:56.631526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0dd8d532-aebb-4b82-8f39-613411fff2b3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:56.633389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0dd8d532-aebb-4b82-8f39-613411fff2b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:56.688042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:56.744757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 655/1294] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:56.773704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8179583-db95-4446-ae55-37c59ae09e17 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:56.822917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:56.911026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8179583-db95-4446-ae55-37c59ae09e17 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:56.954238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c630c0f-e2ef-4afe-8da6-137f4fd835b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:56.955150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c630c0f-e2ef-4afe-8da6-137f4fd835b8 None admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:56.955644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c630c0f-e2ef-4afe-8da6-137f4fd835b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:56.956100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c630c0f-e2ef-4afe-8da6-137f4fd835b8 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:56.956914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c630c0f-e2ef-4afe-8da6-137f4fd835b8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:56.958307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c630c0f-e2ef-4afe-8da6-137f4fd835b8 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:56.981343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 640/1295] 10.222.0.26 () {60 vars in 1141 bytes} [Sun Aug 7 00:59:56 2022] POST /identity/v3/services => generated 246 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 00:59:59.123157 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3b882-3120-4b2a-ac6b-fcf21510a52c None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:59.123748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3b882-3120-4b2a-ac6b-fcf21510a52c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:59.123748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3b882-3120-4b2a-ac6b-fcf21510a52c None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:59.125461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 656/1296] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 00:59:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 00:59:59.131652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:59.132141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:59.132275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:59.151252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 00:59:59.152519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 00:59:59.235989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 641/1297] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 00:59:59 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 07 00:59:59.260470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c3b882-3120-4b2a-ac6b-fcf21510a52c None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:59.381060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c3b882-3120-4b2a-ac6b-fcf21510a52c None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:59.423296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:59.424328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:59.424779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:59.425187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:59.426463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:59.428826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 00:59:59.444910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 00:59:59.447742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 00:59:59.449812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 657/1298] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 00:59:59 2022] GET /identity/v3/services/placement => generated 90 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 00:59:59.455271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:59.484710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:59.609291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dca6ef5-d45b-4c46-9385-9fb917e89499 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:59.656196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:59.657139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:59.657647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:59.658163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:59.659366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] RBAC: Adding query filter params (name=placement) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:59.659904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:59.662617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:59.672665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 642/1299] 10.222.0.26 () {56 vars in 1116 bytes} [Sun Aug 7 00:59:59 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:59.681833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 00:59:59.741286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 00:59:59.853329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7b23c-cde0-4d5a-a184-e241d0e8f648 None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 00:59:59.899606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 00:59:59.900639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 00:59:59.901100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 00:59:59.901683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 00:59:59.902921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] RBAC: Adding query filter params (interface=public, service_id=83611329ec8a476e871ee12cf2551664, region_id=RegionOne) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 00:59:59.903442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 00:59:59.905921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edcbcc4-03c3-496e-af23-bc8708c7d6fc None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 00:59:59.913088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 658/1300] 10.222.0.26 () {56 vars in 1251 bytes} [Sun Aug 7 00:59:59 2022] GET /identity/v3/endpoints?service_id=83611329ec8a476e871ee12cf2551664&interface=public®ion_id=RegionOne => generated 194 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 00:59:59.920165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:00:00.047557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:00:00.070491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fd4814-c7ff-45bf-ac08-cd180b9cf04f None admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:00:00.116594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc677db-5f7d-449d-8c6c-87cc3f3330d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:00:00.117689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc677db-5f7d-449d-8c6c-87cc3f3330d6 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:00:00.118733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc677db-5f7d-449d-8c6c-87cc3f3330d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:00:00.119221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc677db-5f7d-449d-8c6c-87cc3f3330d6 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:00:00.120351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc677db-5f7d-449d-8c6c-87cc3f3330d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:00:00.122825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc677db-5f7d-449d-8c6c-87cc3f3330d6 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:00:00.135183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 643/1301] 10.222.0.26 () {56 vars in 1087 bytes} [Sun Aug 7 00:59:59 2022] GET /identity/v3/services => generated 2191 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:00:02.620050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b4cf1-2705-433d-89c0-4ff615cfe959 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:00:02.620949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b4cf1-2705-433d-89c0-4ff615cfe959 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:00:02.620949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b4cf1-2705-433d-89c0-4ff615cfe959 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:00:02.622053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 659/1302] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:00:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:00:02.627828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:00:02.628517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:00:02.628517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:00:02.652051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:00:02.653349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:00:02.728797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 644/1303] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:00:02 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 07 01:00:02.771375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b4cf1-2705-433d-89c0-4ff615cfe959 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:00:02.904909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:00:02.917115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b4cf1-2705-433d-89c0-4ff615cfe959 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:00:02.963429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:00:02.964214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:00:02.964393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:00:02.964650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] PATH_INFO: `/v3/services/placement` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:00:02.965239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:00:02.966615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 07 01:00:02.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 07 01:00:02.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.get_or_create( Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application with Lock( Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self._enter() Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application return self.creator() Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application created_value = creator( Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: ERROR keystone.server.flask.application Aug 07 01:00:02.980187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 660/1304] 10.222.0.26 () {56 vars in 1117 bytes} [Sun Aug 7 01:00:02 2022] GET /identity/v3/services/placement => generated 90 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 07 01:00:02.985807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:00:03.101780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:00:03.139609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d1b987-cd0b-4dd8-84c4-fb4c386b6f29 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:00:03.193418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:00:03.193924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:00:03.194204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:00:03.194473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] PATH_INFO: `/v3/services` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:00:03.195119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:00:03.195532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:00:03.198531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56de672e-5ad9-4532-8c48-4c32e9f8f4a3 None admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:00:03.208188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 645/1305] 10.222.0.26 () {56 vars in 1116 bytes} [Sun Aug 7 01:00:02 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:00:03.217915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:00:03.416705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f972f009-2d49-4790-880c-b566ab29204d None admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:00:03.475957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aed39cc-54fb-4453-928e-bce0a33cb1bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:00:03.476967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aed39cc-54fb-4453-928e-bce0a33cb1bd None admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:00:03.478001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aed39cc-54fb-4453-928e-bce0a33cb1bd None admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:00:03.478292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aed39cc-54fb-4453-928e-bce0a33cb1bd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:00:03.478995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aed39cc-54fb-4453-928e-bce0a33cb1bd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:00:03.480406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aed39cc-54fb-4453-928e-bce0a33cb1bd None admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:00:03.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 661/1306] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:00:03 2022] POST /identity/v3/endpoints => generated 331 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:00:13.202891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=30 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:00:13.500464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:16.847204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c9660a-0035-4b2d-9048-6c6a2cc4c095 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:16.848155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c9660a-0035-4b2d-9048-6c6a2cc4c095 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:16.848155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c9660a-0035-4b2d-9048-6c6a2cc4c095 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:16.856896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 646/1307] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:16 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:16.861032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552933d1-2f7d-4f09-95f9-743ec35a8697 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:16.861869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552933d1-2f7d-4f09-95f9-743ec35a8697 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:16.861869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552933d1-2f7d-4f09-95f9-743ec35a8697 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:17.028529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-552933d1-2f7d-4f09-95f9-743ec35a8697 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:17.031181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-552933d1-2f7d-4f09-95f9-743ec35a8697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:17.272940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 662/1308] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:17.334732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b0e0fe-f0ba-4f6c-9d94-a903d4577819 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:17.336139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b0e0fe-f0ba-4f6c-9d94-a903d4577819 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:17.337035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b0e0fe-f0ba-4f6c-9d94-a903d4577819 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:17.340685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 647/1309] 10.222.0.26 () {54 vars in 965 bytes} [Sun Aug 7 01:01:17 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:17.348663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082eb8c-0c99-4675-ada7-758f7167dabb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:17.349220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082eb8c-0c99-4675-ada7-758f7167dabb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:17.349896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082eb8c-0c99-4675-ada7-758f7167dabb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:17.506845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a082eb8c-0c99-4675-ada7-758f7167dabb None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:17.508958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a082eb8c-0c99-4675-ada7-758f7167dabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:17.553571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:17.672833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 663/1310] 10.222.0.26 () {58 vars in 1062 bytes} [Sun Aug 7 01:01:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:17.682512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b0e0fe-f0ba-4f6c-9d94-a903d4577819 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:17.684446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f5b0e0fe-f0ba-4f6c-9d94-a903d4577819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:17.872872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b0e0fe-f0ba-4f6c-9d94-a903d4577819 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:18.092191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:18.093325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:18.093934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:18.094340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:18.097899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:18.241226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:18.340750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:18.342430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fed5575-619b-45bf-8c69-e1e96e4d0661 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:18.382049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 648/1311] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 698 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:01:21.460997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97495d02-78e2-4f55-b809-6496411c7b6b None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:21.461823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97495d02-78e2-4f55-b809-6496411c7b6b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:21.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97495d02-78e2-4f55-b809-6496411c7b6b None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:21.464385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 664/1312] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:21.472884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c17474-4e14-48bc-adf9-59a6f5173f50 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:21.473620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c17474-4e14-48bc-adf9-59a6f5173f50 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:21.473620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c17474-4e14-48bc-adf9-59a6f5173f50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:21.580747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-66c17474-4e14-48bc-adf9-59a6f5173f50 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:21.582846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-66c17474-4e14-48bc-adf9-59a6f5173f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:21.728530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 649/1313] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:01:21.812176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806419f1-e342-4f57-b6aa-340bcbad1fe9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:21.812637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806419f1-e342-4f57-b6aa-340bcbad1fe9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:21.813249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806419f1-e342-4f57-b6aa-340bcbad1fe9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:21.814807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 665/1314] 10.222.0.26 () {54 vars in 965 bytes} [Sun Aug 7 01:01:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:01:21.824164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc2f252-9a4c-4660-abe6-dcf1a9aaf745 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:21.824646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc2f252-9a4c-4660-abe6-dcf1a9aaf745 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:21.825013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc2f252-9a4c-4660-abe6-dcf1a9aaf745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:21.839418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:21.930080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6dc2f252-9a4c-4660-abe6-dcf1a9aaf745 None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:21.931382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6dc2f252-9a4c-4660-abe6-dcf1a9aaf745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:22.050201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 650/1315] 10.222.0.26 () {58 vars in 1062 bytes} [Sun Aug 7 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:01:22.063905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806419f1-e342-4f57-b6aa-340bcbad1fe9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:22.066658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-806419f1-e342-4f57-b6aa-340bcbad1fe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:22.260374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806419f1-e342-4f57-b6aa-340bcbad1fe9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:22.393314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:22.398225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:22.398780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:22.399502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:22.400008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:22.403373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:22.618593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:22.621413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d56dd6-0fe7-4fab-8779-6cc043e1b97a service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:22.660099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 666/1316] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 597 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:01:25.926929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed4266-11eb-4503-ae93-0e456a93e91f None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:25.927917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed4266-11eb-4503-ae93-0e456a93e91f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:25.927917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed4266-11eb-4503-ae93-0e456a93e91f None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:25.929226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 651/1317] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:01:25.935434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5206ed-2946-48ef-b362-b84854c4fce6 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:25.936164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5206ed-2946-48ef-b362-b84854c4fce6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:25.936164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5206ed-2946-48ef-b362-b84854c4fce6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:26.026056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ae5206ed-2946-48ef-b362-b84854c4fce6 None None] MFA Rules not processed for user `fb5de2b0ea574c64b443f39117fbe50c`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:26.028466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ae5206ed-2946-48ef-b362-b84854c4fce6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:26.136205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:26.214595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 667/1318] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:25 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:01:26.257479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed4266-11eb-4503-ae93-0e456a93e91f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:26.259091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f0ed4266-11eb-4503-ae93-0e456a93e91f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:26.391981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed4266-11eb-4503-ae93-0e456a93e91f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:26.395250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:26.483329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:26.483895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:26.484327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:26.484620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:26.486321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:26.623848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:26.625445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3258d7-4f20-4874-bac5-fc1ab9284fa8 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:26.649634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 652/1319] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:26 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:01:31.036900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94cebe7-50aa-4fe7-a73e-abbe6387595c None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:31.037759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94cebe7-50aa-4fe7-a73e-abbe6387595c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:31.037913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94cebe7-50aa-4fe7-a73e-abbe6387595c None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:31.040333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 668/1320] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:31.049514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62300f7-331b-4b66-95f9-6075e4c528d7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:31.050183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62300f7-331b-4b66-95f9-6075e4c528d7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:31.050535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62300f7-331b-4b66-95f9-6075e4c528d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:31.140982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:31.183086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a62300f7-331b-4b66-95f9-6075e4c528d7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:31.184722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a62300f7-331b-4b66-95f9-6075e4c528d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:31.329419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 653/1321] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:31.374068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94cebe7-50aa-4fe7-a73e-abbe6387595c None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:31.376694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a94cebe7-50aa-4fe7-a73e-abbe6387595c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:31.530631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94cebe7-50aa-4fe7-a73e-abbe6387595c None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:31.634870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:31.635542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:31.635903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:31.636218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:31.638177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:31.647544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:31.799319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:31.801107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d00fb-7208-4aff-8189-3504870ef340 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:31.823650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 669/1322] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:01:35.279295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdedadd-6fed-4056-9658-bef1bdb4748b None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:35.280151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdedadd-6fed-4056-9658-bef1bdb4748b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:35.280550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdedadd-6fed-4056-9658-bef1bdb4748b None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:35.282755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 654/1323] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:35.291206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac055709-eecc-44c0-b960-a2ecddafa44c None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:35.292033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac055709-eecc-44c0-b960-a2ecddafa44c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:35.292033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac055709-eecc-44c0-b960-a2ecddafa44c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:35.378402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ac055709-eecc-44c0-b960-a2ecddafa44c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:35.380460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ac055709-eecc-44c0-b960-a2ecddafa44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:35.462690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:35.543439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 670/1324] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:35.588979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdedadd-6fed-4056-9658-bef1bdb4748b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:35.590731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-afdedadd-6fed-4056-9658-bef1bdb4748b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:35.726945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdedadd-6fed-4056-9658-bef1bdb4748b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:35.743345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:35.846887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:35.847813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:35.848351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:35.848821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:35.851994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:36.049978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:36.052062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c714f4-77f0-4082-94a9-2c1138471a30 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:36.081487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 655/1325] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:01:38.672016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9cff14-df0c-4ddb-ab69-b06cb7bfacd1 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:38.672904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9cff14-df0c-4ddb-ab69-b06cb7bfacd1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:38.672904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9cff14-df0c-4ddb-ab69-b06cb7bfacd1 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:38.674555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 671/1326] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:38.681033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984fb5d-4879-4740-8b14-ca4c33d4f924 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:38.681933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984fb5d-4879-4740-8b14-ca4c33d4f924 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:38.682113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984fb5d-4879-4740-8b14-ca4c33d4f924 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:38.744138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:38.767496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c984fb5d-4879-4740-8b14-ca4c33d4f924 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:38.768766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c984fb5d-4879-4740-8b14-ca4c33d4f924 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:38.941127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 656/1327] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:38.981307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9cff14-df0c-4ddb-ab69-b06cb7bfacd1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:38.983212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ed9cff14-df0c-4ddb-ab69-b06cb7bfacd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:39.149411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9cff14-df0c-4ddb-ab69-b06cb7bfacd1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:39.275026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:39.276064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:39.276589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:39.277079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:39.280417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:39.307581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:39.428271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:39.430002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae44fb-a318-4e52-b6c7-75bc21e38493 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:39.455493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 672/1328] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:01:41.918389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213d7a3c-3e72-4266-8578-3781e8084ef3 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:41.919270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213d7a3c-3e72-4266-8578-3781e8084ef3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:41.919270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213d7a3c-3e72-4266-8578-3781e8084ef3 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:41.920576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 657/1329] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:01:41.927714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e260741-0e23-42eb-a1ec-063e6fa9d80e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:41.928607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e260741-0e23-42eb-a1ec-063e6fa9d80e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:41.928784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e260741-0e23-42eb-a1ec-063e6fa9d80e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:42.040206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5e260741-0e23-42eb-a1ec-063e6fa9d80e None None] MFA Rules not processed for user `fb5de2b0ea574c64b443f39117fbe50c`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:42.042461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5e260741-0e23-42eb-a1ec-063e6fa9d80e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:42.181873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:42.249316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 673/1330] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:42.317637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213d7a3c-3e72-4266-8578-3781e8084ef3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:42.324783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-213d7a3c-3e72-4266-8578-3781e8084ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:42.498382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213d7a3c-3e72-4266-8578-3781e8084ef3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:42.525359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:42.620785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:42.621570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:42.621859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:42.622139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:42.623857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:42.760666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:42.762045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858737df-8bb7-4be3-9721-9850c632f73d service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:42.785072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 658/1331] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:42 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:01:45.701916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62eb137-dfe4-4151-8ea2-d6fd26b43280 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:45.702798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62eb137-dfe4-4151-8ea2-d6fd26b43280 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:45.702971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62eb137-dfe4-4151-8ea2-d6fd26b43280 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:45.705605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 674/1332] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:45.715119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95218228-06c4-4965-bdb1-6b550bf6e15c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:45.715975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95218228-06c4-4965-bdb1-6b550bf6e15c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:45.716130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95218228-06c4-4965-bdb1-6b550bf6e15c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:45.834770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:45.838740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-95218228-06c4-4965-bdb1-6b550bf6e15c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:45.840941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-95218228-06c4-4965-bdb1-6b550bf6e15c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:45.998971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 659/1333] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:46.061001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62eb137-dfe4-4151-8ea2-d6fd26b43280 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:46.063200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b62eb137-dfe4-4151-8ea2-d6fd26b43280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:46.221334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62eb137-dfe4-4151-8ea2-d6fd26b43280 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:46.360433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:46.361503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:46.361999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:46.362491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:46.366490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:46.422601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:46.539972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:46.541885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a107829-ea1f-43ee-91f6-cab5e42f6dbf service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:46.567543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 675/1334] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:01:48.958009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64124d6-7e64-47e6-a1b9-140935fb8690 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:48.958786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64124d6-7e64-47e6-a1b9-140935fb8690 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:48.958786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64124d6-7e64-47e6-a1b9-140935fb8690 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:48.960225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 660/1335] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:48.966401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2538d-5105-4a2c-82a1-8418be632547 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:48.966982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2538d-5105-4a2c-82a1-8418be632547 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:48.967156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2538d-5105-4a2c-82a1-8418be632547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:49.066829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-dec2538d-5105-4a2c-82a1-8418be632547 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:49.069048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dec2538d-5105-4a2c-82a1-8418be632547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:49.211865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:49.241328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 676/1336] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:49.311924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64124d6-7e64-47e6-a1b9-140935fb8690 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:49.313953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b64124d6-7e64-47e6-a1b9-140935fb8690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:49.471932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64124d6-7e64-47e6-a1b9-140935fb8690 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:49.512197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:49.574069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:49.575135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:49.575603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:49.576058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:49.578819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:49.733075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:49.735051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93c92ab-1897-44ea-b68b-45453b0bd546 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:49.757187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 661/1337] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:01:52.673080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1da27f-eeb0-4952-b5c1-2558a0e560cf None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:52.673914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1da27f-eeb0-4952-b5c1-2558a0e560cf None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:52.674080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1da27f-eeb0-4952-b5c1-2558a0e560cf None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:52.677090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 677/1338] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:52 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:52.683691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869e4586-04fa-42cc-82c4-f06cced61942 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:52.684584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869e4586-04fa-42cc-82c4-f06cced61942 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:52.684745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869e4586-04fa-42cc-82c4-f06cced61942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:52.768864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-869e4586-04fa-42cc-82c4-f06cced61942 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:52.770319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-869e4586-04fa-42cc-82c4-f06cced61942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:52.780625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:52.923948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 662/1339] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:52 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:01:52.986459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1da27f-eeb0-4952-b5c1-2558a0e560cf None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:52.989404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-da1da27f-eeb0-4952-b5c1-2558a0e560cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:53.141508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1da27f-eeb0-4952-b5c1-2558a0e560cf None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:53.274461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:53.275292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:53.275674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:53.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:53.277908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:53.333726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:53.491848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:53.493450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c904a60-f2e1-49e5-8985-b3e4a9ceb3e9 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:53.518007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 678/1340] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:52 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:01:58.985112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d750452-7e99-4960-a4b8-80b7922390ac None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:58.986059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d750452-7e99-4960-a4b8-80b7922390ac None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:58.986059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d750452-7e99-4960-a4b8-80b7922390ac None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:58.989293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 663/1341] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:01:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:01:58.998722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acdb042-40b7-4ce2-b45c-78d256f414d8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:58.999337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acdb042-40b7-4ce2-b45c-78d256f414d8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:58.999710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acdb042-40b7-4ce2-b45c-78d256f414d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:59.111202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9acdb042-40b7-4ce2-b45c-78d256f414d8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:01:59.112947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9acdb042-40b7-4ce2-b45c-78d256f414d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:59.266929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:59.283914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 679/1342] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:01:59.353346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d750452-7e99-4960-a4b8-80b7922390ac None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:01:59.356583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8d750452-7e99-4960-a4b8-80b7922390ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:59.572033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d750452-7e99-4960-a4b8-80b7922390ac None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:01:59.628653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:01:59.708256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:01:59.711582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:01:59.711582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:01:59.711582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:01:59.713605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:01:59.880845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:01:59.882293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aad531e-8876-470a-91b4-7f523f71aabb service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:01:59.905393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 664/1343] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:01:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 554 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:02:03.247385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f2d05-6645-4f0d-b591-1c0252bc7b82 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:03.247966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f2d05-6645-4f0d-b591-1c0252bc7b82 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:03.248437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f2d05-6645-4f0d-b591-1c0252bc7b82 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:03.250871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 680/1344] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:03.258109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a247f9ba-9c61-403a-a7ab-736001250c8e None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:03.265864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a247f9ba-9c61-403a-a7ab-736001250c8e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:03.265864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a247f9ba-9c61-403a-a7ab-736001250c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:03.354593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a247f9ba-9c61-403a-a7ab-736001250c8e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:03.356675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a247f9ba-9c61-403a-a7ab-736001250c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:03.389367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:03.533150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 665/1345] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:03 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:03.575370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f2d05-6645-4f0d-b591-1c0252bc7b82 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:03.577222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-348f2d05-6645-4f0d-b591-1c0252bc7b82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:03.776808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f2d05-6645-4f0d-b591-1c0252bc7b82 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:03.917758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:03.918793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:03.919281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:03.919733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:03.922619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:04.009133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:04.120768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:04.122748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bf7a78-a13d-426a-a360-5d4ed90dbeb8 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:04.154994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 681/1346] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:03 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 580 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:02:10.158559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efdb339-cce2-4640-a309-a8d2ad51ef8a None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:10.159969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efdb339-cce2-4640-a309-a8d2ad51ef8a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:10.160732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efdb339-cce2-4640-a309-a8d2ad51ef8a None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:10.162726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 666/1347] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:10.169800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36540726-d536-4872-96dc-0811200252a2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:10.170703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36540726-d536-4872-96dc-0811200252a2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:10.170703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36540726-d536-4872-96dc-0811200252a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:10.287627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-36540726-d536-4872-96dc-0811200252a2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:10.289302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-36540726-d536-4872-96dc-0811200252a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:10.427833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:10.436485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 682/1348] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:10.501313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efdb339-cce2-4640-a309-a8d2ad51ef8a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:10.504630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3efdb339-cce2-4640-a309-a8d2ad51ef8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:10.684960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3efdb339-cce2-4640-a309-a8d2ad51ef8a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:10.769667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:10.826141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:10.827225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:10.827735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:10.828263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:10.831421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:11.047975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:11.050879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb123142-dfa8-4250-8578-dd4b5547f4bf service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:11.087580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 667/1349] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 587 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:02:17.384650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d813b-8a6a-43b7-bbb1-4299e42fc6c2 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:17.386236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d813b-8a6a-43b7-bbb1-4299e42fc6c2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:17.386980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46d813b-8a6a-43b7-bbb1-4299e42fc6c2 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:17.391832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 683/1350] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:02:17 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:17.399073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa83902-795b-462c-b228-43f92653f667 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:17.399870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa83902-795b-462c-b228-43f92653f667 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:17.400036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa83902-795b-462c-b228-43f92653f667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:17.587196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-0aa83902-795b-462c-b228-43f92653f667 None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:17.588933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0aa83902-795b-462c-b228-43f92653f667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:17.624927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:17.899401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 668/1351] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:02:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 502 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:23.824631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe8f51-4514-4507-a298-fd7940ec0b54 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:23.825206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe8f51-4514-4507-a298-fd7940ec0b54 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:23.825206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe8f51-4514-4507-a298-fd7940ec0b54 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:23.827308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 684/1352] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:02:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:23.834798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed080a3f-0eba-435a-a499-e4c93fd315e6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:23.835588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed080a3f-0eba-435a-a499-e4c93fd315e6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:23.835588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed080a3f-0eba-435a-a499-e4c93fd315e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:23.937761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ed080a3f-0eba-435a-a499-e4c93fd315e6 None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:23.939817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ed080a3f-0eba-435a-a499-e4c93fd315e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:24.042639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:24.102138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 669/1353] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:02:23 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:25.714038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097a80ae-1acf-4d21-b051-423731452389 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:25.714895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097a80ae-1acf-4d21-b051-423731452389 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:25.714895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097a80ae-1acf-4d21-b051-423731452389 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:25.715931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 685/1354] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:02:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:25.722857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64b3735-ed32-45c5-95f4-22d97272da7f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:25.723428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64b3735-ed32-45c5-95f4-22d97272da7f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:25.723734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64b3735-ed32-45c5-95f4-22d97272da7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:25.885163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b64b3735-ed32-45c5-95f4-22d97272da7f None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:25.887164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b64b3735-ed32-45c5-95f4-22d97272da7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:26.110419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 670/1355] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:02:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 390 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:30.116127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8205751-9931-405f-8465-1e9bf2dd2c1e None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:30.116809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8205751-9931-405f-8465-1e9bf2dd2c1e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:30.117064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8205751-9931-405f-8465-1e9bf2dd2c1e None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:30.119432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 686/1356] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:30.129773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b78883-9802-43e4-93ed-da48fc46f0fa None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:30.131449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b78883-9802-43e4-93ed-da48fc46f0fa None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:30.131937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b78883-9802-43e4-93ed-da48fc46f0fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:30.226293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:30.279731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c4b78883-9802-43e4-93ed-da48fc46f0fa None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:30.281987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c4b78883-9802-43e4-93ed-da48fc46f0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:30.444765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 671/1357] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:30.514817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ebc09a-c0b5-4dab-b583-2cc57c090cb8 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:30.515531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ebc09a-c0b5-4dab-b583-2cc57c090cb8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:30.515955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ebc09a-c0b5-4dab-b583-2cc57c090cb8 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:30.517839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 687/1358] 10.222.0.26 () {54 vars in 962 bytes} [Sun Aug 7 01:02:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:30.529510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dd003e-6238-4ae6-91b4-63ca4203026d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:30.530124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dd003e-6238-4ae6-91b4-63ca4203026d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:30.530707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dd003e-6238-4ae6-91b4-63ca4203026d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:30.655820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d2dd003e-6238-4ae6-91b4-63ca4203026d None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:30.657255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d2dd003e-6238-4ae6-91b4-63ca4203026d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:30.726970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:30.830387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 672/1359] 10.222.0.26 () {58 vars in 1059 bytes} [Sun Aug 7 01:02:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:30.840051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ebc09a-c0b5-4dab-b583-2cc57c090cb8 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:30.841915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-37ebc09a-c0b5-4dab-b583-2cc57c090cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:31.064494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ebc09a-c0b5-4dab-b583-2cc57c090cb8 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:31.208885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:31.262411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:31.263062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:31.263347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:31.263664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:31.265462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:31.398820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:31.401854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c3a8be-4361-40ea-959a-aed776a361c9 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:31.425761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 688/1360] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 586 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:02:34.136937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195e08cb-7390-476f-afae-a15c109ee218 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:34.137552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195e08cb-7390-476f-afae-a15c109ee218 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:34.138013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195e08cb-7390-476f-afae-a15c109ee218 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:34.141303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 673/1361] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:34.151203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359723fb-db89-494e-be96-69f39c76c467 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:34.151757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359723fb-db89-494e-be96-69f39c76c467 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:34.152168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359723fb-db89-494e-be96-69f39c76c467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:34.252678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-359723fb-db89-494e-be96-69f39c76c467 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:34.253885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-359723fb-db89-494e-be96-69f39c76c467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:34.314382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:34.468408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 689/1362] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:34.550103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195e08cb-7390-476f-afae-a15c109ee218 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:34.551716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-195e08cb-7390-476f-afae-a15c109ee218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:34.805064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195e08cb-7390-476f-afae-a15c109ee218 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:34.819443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:34.986670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:34.987286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:34.987579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:34.987850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:34.989730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:35.117872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:35.120047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783e4bd9-6204-4cf2-b0cf-bae6737dc591 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:35.144540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 674/1363] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 595 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:02:37.740110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d5534-2ec9-4fe0-8d31-d99f49a76414 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:37.740682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d5534-2ec9-4fe0-8d31-d99f49a76414 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:37.741118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979d5534-2ec9-4fe0-8d31-d99f49a76414 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:37.743848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 690/1364] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:37.750291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b516cc5-5fe3-46ea-814a-ea74e8335f86 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:37.751683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b516cc5-5fe3-46ea-814a-ea74e8335f86 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:37.751683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b516cc5-5fe3-46ea-814a-ea74e8335f86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:37.778832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:37.900894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9b516cc5-5fe3-46ea-814a-ea74e8335f86 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:37.903078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9b516cc5-5fe3-46ea-814a-ea74e8335f86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:38.101563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 675/1365] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:38.195261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a9eb0-1595-4ed5-a97a-767da2d1769f None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:38.195737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a9eb0-1595-4ed5-a97a-767da2d1769f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:38.196198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a9eb0-1595-4ed5-a97a-767da2d1769f None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:38.198597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 691/1366] 10.222.0.26 () {54 vars in 962 bytes} [Sun Aug 7 01:02:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:38.209300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d878dd58-0277-4d3e-abf9-4c135d4c1967 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:38.209825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d878dd58-0277-4d3e-abf9-4c135d4c1967 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:38.210370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d878dd58-0277-4d3e-abf9-4c135d4c1967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:38.362327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d878dd58-0277-4d3e-abf9-4c135d4c1967 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:38.364435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d878dd58-0277-4d3e-abf9-4c135d4c1967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:38.433962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:38.611060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 676/1367] 10.222.0.26 () {58 vars in 1059 bytes} [Sun Aug 7 01:02:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 403 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:38.628164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0a9eb0-1595-4ed5-a97a-767da2d1769f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:38.630788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4b0a9eb0-1595-4ed5-a97a-767da2d1769f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:38.863071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0a9eb0-1595-4ed5-a97a-767da2d1769f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:38.890936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:39.043010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:39.044003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:39.044558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:39.044842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:39.046723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:39.182191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:39.184155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd20d5-6255-4872-8ef0-dc4198657afc service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:39.219983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 692/1368] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 593 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:02:41.681720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d1926-f0ec-4580-8dd3-0d3af5a6e3b4 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:41.682391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d1926-f0ec-4580-8dd3-0d3af5a6e3b4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:41.682488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927d1926-f0ec-4580-8dd3-0d3af5a6e3b4 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:41.690679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a695f-4486-485c-9f44-69ab6f90de35 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:41.691313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a695f-4486-485c-9f44-69ab6f90de35 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:41.691313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4a695f-4486-485c-9f44-69ab6f90de35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:41.696548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 677/1369] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:41 2022] GET /identity => generated 270 bytes in 16 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:41.752842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:41.777161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7e4a695f-4486-485c-9f44-69ab6f90de35 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:41.778483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7e4a695f-4486-485c-9f44-69ab6f90de35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:41.948904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 693/1370] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:02:42.152073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53e73b-d55e-446e-8ae5-5bfb1d456465 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:42.152533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53e73b-d55e-446e-8ae5-5bfb1d456465 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:42.152740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53e73b-d55e-446e-8ae5-5bfb1d456465 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:42.154962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 678/1371] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:42.161580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:42.161849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:42.162125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:42.296463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:42.299598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:42.374147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606124a9-243c-4413-bec9-2fdc5750c136 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:42.375045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606124a9-243c-4413-bec9-2fdc5750c136 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:42.375853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606124a9-243c-4413-bec9-2fdc5750c136 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:42.378728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 679/1372] 10.222.0.26 () {54 vars in 964 bytes} [Sun Aug 7 01:02:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:42.386073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:42.386968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:42.387704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:42.421954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:42.485402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 694/1373] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:42.536314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:42.538731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:42.547424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] MFA Rules not processed for user `f2a3c404c5f840368281af8532b7b46b`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:42.549757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:42.714788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 680/1374] 10.222.0.26 () {58 vars in 1061 bytes} [Sun Aug 7 01:02:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:42.725868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:42.736434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2f4aba-60d0-419c-bbec-1828d73a5a5c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:42.741335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:42.771464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2297e6e6-b198-4c2a-a966-96cf81f7781f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:42.890191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:42.890800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:42.891854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:42.893899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:42.893899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:42.897147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:42.979621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:42.980910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:42.981539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:42.982087 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:42.985933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:43.108373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:43.112716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2afde-edac-45ce-a656-03f9995f5f50 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:43.153699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 681/1375] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:02:43.175035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:43.178425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93487f06-2851-4ba9-8976-974b718f040e service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:43.219030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 695/1376] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 684 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:02:45.768641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539dd6c-a9dd-4d82-85e5-876dfb8f9bb1 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:45.769428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539dd6c-a9dd-4d82-85e5-876dfb8f9bb1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:45.769428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c539dd6c-a9dd-4d82-85e5-876dfb8f9bb1 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:45.770689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 682/1377] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:45.777168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4523c5f-0e50-492b-985a-9ff39dd37851 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:45.777729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4523c5f-0e50-492b-985a-9ff39dd37851 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:45.778102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4523c5f-0e50-492b-985a-9ff39dd37851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:45.878316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a4523c5f-0e50-492b-985a-9ff39dd37851 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:45.880148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a4523c5f-0e50-492b-985a-9ff39dd37851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:45.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:46.053276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 696/1378] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:46.115442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539dd6c-a9dd-4d82-85e5-876dfb8f9bb1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:46.118448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c539dd6c-a9dd-4d82-85e5-876dfb8f9bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:46.192797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:46.406583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c539dd6c-a9dd-4d82-85e5-876dfb8f9bb1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:46.565176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:46.576227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:46.576699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:46.576947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:46.577222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:46.579086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:46.715096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:46.718043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7906bd2-4ef4-4a1a-ad6d-cd1bb9e6d1e5 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:46.744210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 683/1379] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 630 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:02:49.864971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef58095-b537-418e-9969-4486b953c929 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:49.865535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef58095-b537-418e-9969-4486b953c929 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:49.866140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef58095-b537-418e-9969-4486b953c929 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:49.868127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 697/1380] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:02:49.873949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9538034-5b78-4aff-85f0-db49e8738602 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:49.874464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9538034-5b78-4aff-85f0-db49e8738602 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:49.874669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9538034-5b78-4aff-85f0-db49e8738602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:49.972937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e9538034-5b78-4aff-85f0-db49e8738602 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:49.974470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e9538034-5b78-4aff-85f0-db49e8738602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:50.060696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:50.104056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 684/1381] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:50.153364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef58095-b537-418e-9969-4486b953c929 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:50.154908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1ef58095-b537-418e-9969-4486b953c929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:50.349873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef58095-b537-418e-9969-4486b953c929 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:50.366049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:50.496973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:50.498356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:50.498910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:50.500496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:50.503168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:50.626829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:50.629872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d18031-56ff-4ce9-ab26-41a15068cab1 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:50.666877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 698/1382] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 514 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:02:53.914241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea4bbc4-0522-4d11-a63b-134491cef3ec None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:53.914949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea4bbc4-0522-4d11-a63b-134491cef3ec None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:53.914949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea4bbc4-0522-4d11-a63b-134491cef3ec None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:53.916215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 685/1383] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:53.927213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c056835e-96a9-40de-99c7-407accc82801 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:53.928388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c056835e-96a9-40de-99c7-407accc82801 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:53.928388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c056835e-96a9-40de-99c7-407accc82801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:53.981277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:54.059884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c056835e-96a9-40de-99c7-407accc82801 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:54.061963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c056835e-96a9-40de-99c7-407accc82801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:54.363427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 699/1384] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:54.426089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea4bbc4-0522-4d11-a63b-134491cef3ec None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:54.428118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4ea4bbc4-0522-4d11-a63b-134491cef3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:54.688792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea4bbc4-0522-4d11-a63b-134491cef3ec None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:54.765019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:54.891222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:54.892161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:54.892680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:54.893201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:54.896182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:55.090489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:55.095321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6778f66-f60d-4af5-9b93-cbf60e9f124c service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:55.133562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 686/1385] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 708 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:02:57.755236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3caf00-fc0b-4466-a56e-2eaab225289a None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:57.755815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3caf00-fc0b-4466-a56e-2eaab225289a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:57.755815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3caf00-fc0b-4466-a56e-2eaab225289a None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:57.757158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 700/1386] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:02:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:02:57.763604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902a72d-7165-41ce-aa62-93d7e721d8cb None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:57.764345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902a72d-7165-41ce-aa62-93d7e721d8cb None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:57.764345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902a72d-7165-41ce-aa62-93d7e721d8cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:57.876995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2902a72d-7165-41ce-aa62-93d7e721d8cb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:02:57.878115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2902a72d-7165-41ce-aa62-93d7e721d8cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:57.886580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:58.029752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 687/1387] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:02:58.085833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3caf00-fc0b-4466-a56e-2eaab225289a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:02:58.088679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9e3caf00-fc0b-4466-a56e-2eaab225289a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:58.247528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:58.339756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3caf00-fc0b-4466-a56e-2eaab225289a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:02:58.502894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:02:58.503416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:02:58.503690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:02:58.503957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:02:58.505668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:02:58.609874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:02:58.663170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:02:58.665686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aaff01a-3947-464e-9b60-623d0342ecce service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:02:58.697723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 701/1388] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:02:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 613 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:01.925340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f19505-c730-431f-9734-60914e0b5b0c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:01.926279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f19505-c730-431f-9734-60914e0b5b0c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:01.926279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f19505-c730-431f-9734-60914e0b5b0c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:01.929023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 688/1389] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:03:01 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:01.940002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:01.940727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:01.941037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:01.970460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dea1fd-b2ea-46fc-b62f-15be4047c71d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:01.970779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dea1fd-b2ea-46fc-b62f-15be4047c71d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:01.971227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dea1fd-b2ea-46fc-b62f-15be4047c71d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:01.972503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 689/1390] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:03:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:03:01.980447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8839c3-db56-459e-959a-7a325b49ad08 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:01.980785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8839c3-db56-459e-959a-7a325b49ad08 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:01.980965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8839c3-db56-459e-959a-7a325b49ad08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:02.058052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:02.059737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:02.079139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-bd8839c3-db56-459e-959a-7a325b49ad08 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:02.080604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bd8839c3-db56-459e-959a-7a325b49ad08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:02.210620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 702/1391] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:03:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:02.214004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:02.231202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 690/1392] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:03:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:02.263805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:02.266578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:02.307190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24bf18-03b3-418f-b181-6bdffbd76df4 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:02.307500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24bf18-03b3-418f-b181-6bdffbd76df4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:02.307788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24bf18-03b3-418f-b181-6bdffbd76df4 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:02.309302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 691/1393] 10.222.0.26 () {54 vars in 966 bytes} [Sun Aug 7 01:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:02.318659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:02.319343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:02.319800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:02.329527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:02.436383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] MFA Rules not processed for user `7bd8b2ff3ce14290ac0cf476021e3496`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:02.438208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:02.464578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e904a2-0550-4ae4-a24e-7e0d361fe099 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:02.572537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 692/1394] 10.222.0.26 () {58 vars in 1063 bytes} [Sun Aug 7 01:03:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:03:02.582551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:02.591808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8601eff9-e79d-4117-b2fb-1d2263366bfe None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:02.661286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:02.662071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:02.662356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:02.662646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:02.664532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:02.713644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:02.727950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:02.728714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:02.729135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:02.729601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:02.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:02.735802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:02.849979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:02.853461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d7ef2c-98d5-478b-8b3a-f5d7dc2c01f0 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:02.884591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 703/1395] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 622 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:02.930430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:02.932064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10533c0c-bd2c-40a8-91a9-85db3848ba49 service glance] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:02.959072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 693/1396] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:03:03.910372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fa91d8-44b6-4cac-856d-3dea354ca682 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:03.911181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fa91d8-44b6-4cac-856d-3dea354ca682 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:03.911437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fa91d8-44b6-4cac-856d-3dea354ca682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:04.043660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-34fa91d8-44b6-4cac-856d-3dea354ca682 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:04.046004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-34fa91d8-44b6-4cac-856d-3dea354ca682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:04.231960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 704/1397] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:03:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:04.300401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6734c159-254e-4f92-9ed0-d26e6a51a722 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:04.301233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6734c159-254e-4f92-9ed0-d26e6a51a722 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:04.301233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6734c159-254e-4f92-9ed0-d26e6a51a722 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:04.303597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 694/1398] 10.222.0.26 () {54 vars in 943 bytes} [Sun Aug 7 01:03:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:04.314001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74185282-9899-4ef4-8740-7ed13aab3d2c None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:04.314702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74185282-9899-4ef4-8740-7ed13aab3d2c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:04.315231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74185282-9899-4ef4-8740-7ed13aab3d2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:04.368099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:04.452632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-74185282-9899-4ef4-8740-7ed13aab3d2c None None] MFA Rules not processed for user `7342064620f34dabb93dd10f92f444a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:04.454000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-74185282-9899-4ef4-8740-7ed13aab3d2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:04.692919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 705/1399] 10.222.0.26 () {58 vars in 1040 bytes} [Sun Aug 7 01:03:04 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 382 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:04.706690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6734c159-254e-4f92-9ed0-d26e6a51a722 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:04.709665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6734c159-254e-4f92-9ed0-d26e6a51a722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:04.861829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:04.919375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6734c159-254e-4f92-9ed0-d26e6a51a722 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:05.113440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:05.114206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:05.114496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:05.114775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:05.116627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:05.254877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:05.257997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5794a4b7-3e32-427f-ada3-59f6b1ae5771 service swift] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:05.274372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 695/1400] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:03:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 569 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:03:05.773163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583b7600-e752-478d-a4fc-49778f6fb27f None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:05.773934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583b7600-e752-478d-a4fc-49778f6fb27f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:05.773934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583b7600-e752-478d-a4fc-49778f6fb27f None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:05.775192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 706/1401] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:03:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:05.782773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633392a0-3ba9-4eaa-af39-b7ad61ac392b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:05.783589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633392a0-3ba9-4eaa-af39-b7ad61ac392b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:05.783589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633392a0-3ba9-4eaa-af39-b7ad61ac392b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:05.796031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:05.984399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-633392a0-3ba9-4eaa-af39-b7ad61ac392b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:05.986724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-633392a0-3ba9-4eaa-af39-b7ad61ac392b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:06.155072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 696/1402] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:03:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:06.229422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583b7600-e752-478d-a4fc-49778f6fb27f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:06.231075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-583b7600-e752-478d-a4fc-49778f6fb27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:06.351741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:06.467601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583b7600-e752-478d-a4fc-49778f6fb27f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:06.680734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:06.681767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:06.682230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:06.682665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:06.684672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:06.764411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:06.857581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:06.860093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a18c8b-ffea-46a5-a9f9-0748f48f9f37 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:06.885807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 707/1403] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 657 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:09.813220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0225e0-567d-4f6f-89c1-0848e682023c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:09.814025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0225e0-567d-4f6f-89c1-0848e682023c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:09.814025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0225e0-567d-4f6f-89c1-0848e682023c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:09.815071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 697/1404] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:03:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:09.821973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38adabc3-34f8-4112-9924-9729845b45db None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:09.823433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38adabc3-34f8-4112-9924-9729845b45db None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:09.823911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38adabc3-34f8-4112-9924-9729845b45db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:09.929429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-38adabc3-34f8-4112-9924-9729845b45db None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:09.930676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-38adabc3-34f8-4112-9924-9729845b45db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:10.060254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 708/1405] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:03:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:03:10.134398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0225e0-567d-4f6f-89c1-0848e682023c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:10.136120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3b0225e0-567d-4f6f-89c1-0848e682023c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:10.307377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:10.406785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0225e0-567d-4f6f-89c1-0848e682023c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:10.660064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:10.663445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:10.663956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:10.667076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:10.670774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:10.772042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:10.846254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:10.849638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a1210d-4e6d-47bd-8b9e-6241bf37db34 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:10.881615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 698/1406] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 748 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:13.566072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a95b6-6f5b-48cf-917d-35b2944874c1 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:13.567700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a95b6-6f5b-48cf-917d-35b2944874c1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:13.568478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438a95b6-6f5b-48cf-917d-35b2944874c1 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:13.578407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 709/1407] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:03:13 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:13.580893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40026d77-c56f-4d43-ada6-b41cd1ba81cc None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:13.581715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40026d77-c56f-4d43-ada6-b41cd1ba81cc None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:13.582158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40026d77-c56f-4d43-ada6-b41cd1ba81cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:13.670884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-40026d77-c56f-4d43-ada6-b41cd1ba81cc None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:13.673361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-40026d77-c56f-4d43-ada6-b41cd1ba81cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:13.880531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 699/1408] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:03:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:03:13.945935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a95b6-6f5b-48cf-917d-35b2944874c1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:13.947546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-438a95b6-6f5b-48cf-917d-35b2944874c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:13.977300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:14.198522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438a95b6-6f5b-48cf-917d-35b2944874c1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:14.369435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:14.434553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:14.436145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:14.436673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:14.437177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:14.439279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:14.593887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:14.597042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bed7e7a-bd5b-4175-bdad-c3da1b62597e service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:14.631429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 710/1409] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 686 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:03:23.871822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:24.627526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=55 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:31.218857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5b33ff-0117-4dd2-95ea-0791fbd862a3 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:31.219786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5b33ff-0117-4dd2-95ea-0791fbd862a3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:31.219786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5b33ff-0117-4dd2-95ea-0791fbd862a3 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:31.220940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 700/1410] 10.222.0.26 () {54 vars in 918 bytes} [Sun Aug 7 01:03:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:31.227128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0dbd09-4e52-4e0e-8aaa-dd213a868e1b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:31.227691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0dbd09-4e52-4e0e-8aaa-dd213a868e1b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:31.227691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0dbd09-4e52-4e0e-8aaa-dd213a868e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:31.358885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cb0dbd09-4e52-4e0e-8aaa-dd213a868e1b None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:31.360908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cb0dbd09-4e52-4e0e-8aaa-dd213a868e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:31.491237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 711/1411] 10.222.0.26 () {58 vars in 1015 bytes} [Sun Aug 7 01:03:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:03:31.509567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef94416-fdf6-4c21-a122-32e4b982cc42 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:31.509900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef94416-fdf6-4c21-a122-32e4b982cc42 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:31.510214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef94416-fdf6-4c21-a122-32e4b982cc42 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:31.511799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 701/1412] 10.222.0.26 () {54 vars in 918 bytes} [Sun Aug 7 01:03:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:31.518012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d04b87-2f9a-4aaf-bea7-04822b7545ec None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:31.518357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d04b87-2f9a-4aaf-bea7-04822b7545ec None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:31.518769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d04b87-2f9a-4aaf-bea7-04822b7545ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:31.651627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:31.655823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-98d04b87-2f9a-4aaf-bea7-04822b7545ec None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:31.657138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-98d04b87-2f9a-4aaf-bea7-04822b7545ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:31.822872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 712/1413] 10.222.0.26 () {58 vars in 1015 bytes} [Sun Aug 7 01:03:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:34.820185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943398c3-9ff5-4763-b7d1-9be90911d1e8 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:34.820906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943398c3-9ff5-4763-b7d1-9be90911d1e8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:34.820906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943398c3-9ff5-4763-b7d1-9be90911d1e8 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:34.822239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 702/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:03:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:34.828627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461fe067-ded4-4aa4-94c8-447c7fa5ffdb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:34.829527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461fe067-ded4-4aa4-94c8-447c7fa5ffdb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:34.829527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461fe067-ded4-4aa4-94c8-447c7fa5ffdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:34.914241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-461fe067-ded4-4aa4-94c8-447c7fa5ffdb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:34.915893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-461fe067-ded4-4aa4-94c8-447c7fa5ffdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:35.003320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:35.071093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 713/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:03:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:03:35.117538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943398c3-9ff5-4763-b7d1-9be90911d1e8 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:35.119195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-943398c3-9ff5-4763-b7d1-9be90911d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:35.339837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943398c3-9ff5-4763-b7d1-9be90911d1e8 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:35.453894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:35.534712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:35.535324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:35.535603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:35.535875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:35.537905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:35.666652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:35.668268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0aa58-6e48-44dd-a314-f0dc1895b3c2 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:35.689907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 703/1416] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:36.842908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0fca01-2600-48ef-a0b8-331be61dadc6 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:36.843730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0fca01-2600-48ef-a0b8-331be61dadc6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:36.843939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0fca01-2600-48ef-a0b8-331be61dadc6 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:36.846510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 714/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:36.855447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b92dfa-fce7-4435-b001-297bd0e8563c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:36.856295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b92dfa-fce7-4435-b001-297bd0e8563c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:36.856484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b92dfa-fce7-4435-b001-297bd0e8563c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:36.964327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-18b92dfa-fce7-4435-b001-297bd0e8563c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:36.966621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-18b92dfa-fce7-4435-b001-297bd0e8563c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:37.014637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:37.137089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 704/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:03:37.320033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0fca01-2600-48ef-a0b8-331be61dadc6 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:37.322386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2e0fca01-2600-48ef-a0b8-331be61dadc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:37.526397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0fca01-2600-48ef-a0b8-331be61dadc6 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:37.581070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:37.681461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:37.682109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:37.682389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:37.682661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:37.684485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:37.815362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:37.817147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e556de-1093-476a-bba1-438e694181d4 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:37.843960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 715/1419] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:38.841305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deabaad9-37ac-4b5b-bb95-e9bd82fb4049 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:38.841952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deabaad9-37ac-4b5b-bb95-e9bd82fb4049 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:38.841952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deabaad9-37ac-4b5b-bb95-e9bd82fb4049 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:38.843312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 705/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:03:38.850749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15dc1b-e14d-4d4b-a46b-98de7cbb9929 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:38.851577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15dc1b-e14d-4d4b-a46b-98de7cbb9929 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:38.851577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15dc1b-e14d-4d4b-a46b-98de7cbb9929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:38.939888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:38.971333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2b15dc1b-e14d-4d4b-a46b-98de7cbb9929 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:38.972804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2b15dc1b-e14d-4d4b-a46b-98de7cbb9929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:39.112995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 716/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:39.278133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deabaad9-37ac-4b5b-bb95-e9bd82fb4049 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:39.281637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-deabaad9-37ac-4b5b-bb95-e9bd82fb4049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:39.495184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deabaad9-37ac-4b5b-bb95-e9bd82fb4049 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:39.515498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:39.646092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:39.646858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:39.647234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:39.647514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:39.649391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:39.771593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:39.774683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a60675b-d2ad-4c5f-8869-c2e20e294cf9 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:39.809708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 706/1422] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:40.740784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d043d-dc1b-4241-8e2a-59bd4656859d None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:40.741372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d043d-dc1b-4241-8e2a-59bd4656859d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:40.741651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d043d-dc1b-4241-8e2a-59bd4656859d None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:40.744007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 717/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:40.751281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4006fa-6fcf-43ac-9039-bb02a1b879a3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:40.751921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4006fa-6fcf-43ac-9039-bb02a1b879a3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:40.751921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4006fa-6fcf-43ac-9039-bb02a1b879a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:40.779202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:40.853955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2a4006fa-6fcf-43ac-9039-bb02a1b879a3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:40.855166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2a4006fa-6fcf-43ac-9039-bb02a1b879a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:40.977042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 707/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:41.139929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d043d-dc1b-4241-8e2a-59bd4656859d None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:41.141529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b29d043d-dc1b-4241-8e2a-59bd4656859d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:41.266512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:41.309466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d043d-dc1b-4241-8e2a-59bd4656859d None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:41.433238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:41.433782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:41.434176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:41.434458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:41.437439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:41.530638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:41.547921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:41.550003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6450d2c-4aa2-4bf9-88b5-da402d22720c service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:41.574156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 718/1425] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:42.511540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170abcf5-e4ed-42e5-bfec-5799979f2dde None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:42.512467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170abcf5-e4ed-42e5-bfec-5799979f2dde None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:42.512467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170abcf5-e4ed-42e5-bfec-5799979f2dde None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:42.513543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 708/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:42.519228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f1b059-3aff-4fc1-afeb-14a1efa94ccd None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:42.519850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f1b059-3aff-4fc1-afeb-14a1efa94ccd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:42.519850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f1b059-3aff-4fc1-afeb-14a1efa94ccd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:42.617519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-78f1b059-3aff-4fc1-afeb-14a1efa94ccd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:42.618807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-78f1b059-3aff-4fc1-afeb-14a1efa94ccd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:42.743629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 719/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:42.907333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170abcf5-e4ed-42e5-bfec-5799979f2dde None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:42.908904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-170abcf5-e4ed-42e5-bfec-5799979f2dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:43.037962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:43.106385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170abcf5-e4ed-42e5-bfec-5799979f2dde None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:43.261661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:43.262204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:43.262476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:43.262747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:43.264494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:43.324442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:43.374914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:43.376567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0518043-b685-44cd-8d62-e757e352678b service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:43.397370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 709/1428] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:03:44.523151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11665ad-2ea1-4e73-b1b6-80a562c423d9 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:44.523745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11665ad-2ea1-4e73-b1b6-80a562c423d9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:44.523745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11665ad-2ea1-4e73-b1b6-80a562c423d9 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:44.525319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 720/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:44.532026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54b16a7-9a47-4c88-8762-54d7400ddfce None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:44.532605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54b16a7-9a47-4c88-8762-54d7400ddfce None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:44.532605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54b16a7-9a47-4c88-8762-54d7400ddfce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:44.620977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e54b16a7-9a47-4c88-8762-54d7400ddfce None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:44.622232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e54b16a7-9a47-4c88-8762-54d7400ddfce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:44.729834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 710/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:44.881888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11665ad-2ea1-4e73-b1b6-80a562c423d9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:44.883451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f11665ad-2ea1-4e73-b1b6-80a562c423d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:44.937178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:45.041975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11665ad-2ea1-4e73-b1b6-80a562c423d9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:45.158874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:45.159416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:45.159692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:45.159958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:45.161630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:45.170037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:45.261182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:45.262798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181680cf-5933-494a-acbf-72624e3c4900 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:45.284807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 721/1431] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:46.108802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77587412-8c94-40bc-85cd-6260700dcdfa None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:46.109346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77587412-8c94-40bc-85cd-6260700dcdfa None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:46.109346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77587412-8c94-40bc-85cd-6260700dcdfa None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:46.110764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 711/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:46.116205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f1c290-33c2-4af4-97c3-2ea3d8445e85 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:46.116754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f1c290-33c2-4af4-97c3-2ea3d8445e85 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:46.116914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f1c290-33c2-4af4-97c3-2ea3d8445e85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:46.203178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c5f1c290-33c2-4af4-97c3-2ea3d8445e85 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:46.204510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c5f1c290-33c2-4af4-97c3-2ea3d8445e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:46.281149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:46.314241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 722/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:46.468998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77587412-8c94-40bc-85cd-6260700dcdfa None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:46.470498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-77587412-8c94-40bc-85cd-6260700dcdfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:46.490404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:46.636497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77587412-8c94-40bc-85cd-6260700dcdfa None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:46.735234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:46.768135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:46.768791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:46.769075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:46.769347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:46.771037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:46.885742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:46.887374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc0b14c-be8b-443b-8ead-72371517d437 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:46.907746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 712/1434] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:03:47.743711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2611d09-980f-4775-b777-c3d5747a013f None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:47.744398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2611d09-980f-4775-b777-c3d5747a013f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:47.744398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2611d09-980f-4775-b777-c3d5747a013f None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:47.745626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 723/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:47.750578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a309660-d491-45f0-bd7e-5e8b486cdd69 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:47.751026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a309660-d491-45f0-bd7e-5e8b486cdd69 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:47.751136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a309660-d491-45f0-bd7e-5e8b486cdd69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:47.827221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6a309660-d491-45f0-bd7e-5e8b486cdd69 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:47.828369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6a309660-d491-45f0-bd7e-5e8b486cdd69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:47.873714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:47.931844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 713/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:48.093990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2611d09-980f-4775-b777-c3d5747a013f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:48.095480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c2611d09-980f-4775-b777-c3d5747a013f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:48.261373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2611d09-980f-4775-b777-c3d5747a013f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:48.333951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:48.404954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:48.405473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:48.405755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:48.406031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:48.407650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:48.506327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:48.507938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0c48e4-e075-40b4-b9e4-cd94b641d580 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:48.529103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 724/1437] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:49.442089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38426715-a7df-4dd3-8dfb-a2b6893a48cf None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:49.442664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38426715-a7df-4dd3-8dfb-a2b6893a48cf None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:49.442664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38426715-a7df-4dd3-8dfb-a2b6893a48cf None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:49.443988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 714/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:03:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:49.449132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c04464-703f-4d7a-81e5-08d151800349 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:49.449609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c04464-703f-4d7a-81e5-08d151800349 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:49.449717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c04464-703f-4d7a-81e5-08d151800349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:49.534035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e6c04464-703f-4d7a-81e5-08d151800349 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:49.535223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e6c04464-703f-4d7a-81e5-08d151800349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:49.537422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:49.637080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 725/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:03:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:49.793308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38426715-a7df-4dd3-8dfb-a2b6893a48cf None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:49.795051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-38426715-a7df-4dd3-8dfb-a2b6893a48cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:50.075737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38426715-a7df-4dd3-8dfb-a2b6893a48cf None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:50.103344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:50.190766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:50.191376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:50.191652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:50.191918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:50.193570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:50.287674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:50.289326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e86a27-ecc3-4dba-8ce5-759b5e78dc3a service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:50.310351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 715/1440] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:52.235076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07476790-67ab-4f60-b120-936ec2f320d3 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:52.235674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07476790-67ab-4f60-b120-936ec2f320d3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:52.235674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07476790-67ab-4f60-b120-936ec2f320d3 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:52.237137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 726/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:03:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:52.242980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a36c65-3c49-44be-9042-1d9ff4489e35 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:52.243467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a36c65-3c49-44be-9042-1d9ff4489e35 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:52.243574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a36c65-3c49-44be-9042-1d9ff4489e35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:52.288552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:52.317978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-39a36c65-3c49-44be-9042-1d9ff4489e35 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:52.319245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-39a36c65-3c49-44be-9042-1d9ff4489e35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:52.430902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 716/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:03:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:52.485256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07476790-67ab-4f60-b120-936ec2f320d3 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:52.486747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-07476790-67ab-4f60-b120-936ec2f320d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:52.594257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07476790-67ab-4f60-b120-936ec2f320d3 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:52.627729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:52.671316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:52.671922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:52.672206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:52.672523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:52.674138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:52.794207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:52.795807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c038f0c-f183-4358-9eb5-19c8094b7a02 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:52.818556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 727/1443] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:55.039088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e33837b-9ae8-4e54-8b1d-bbe884f0d2af None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:55.039687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e33837b-9ae8-4e54-8b1d-bbe884f0d2af None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:55.039687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e33837b-9ae8-4e54-8b1d-bbe884f0d2af None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:55.041041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 717/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:03:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:55.046864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdabc160-2d69-4343-8c22-e78ee6762de1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:55.047396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdabc160-2d69-4343-8c22-e78ee6762de1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:55.047396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdabc160-2d69-4343-8c22-e78ee6762de1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:55.128920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cdabc160-2d69-4343-8c22-e78ee6762de1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:55.130162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cdabc160-2d69-4343-8c22-e78ee6762de1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:55.145971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:55.254008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 728/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:03:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:55.293179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e33837b-9ae8-4e54-8b1d-bbe884f0d2af None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:55.294679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4e33837b-9ae8-4e54-8b1d-bbe884f0d2af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:55.402869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e33837b-9ae8-4e54-8b1d-bbe884f0d2af None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:55.415182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:55.479032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:55.479644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:55.479919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:55.480184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:55.481858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:55.593887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:55.595407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efad65d-8eef-440c-a741-5653127d35b6 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:55.614811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 718/1446] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:03:58.050500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc5e87-0161-4883-bd88-78f6f4c9679c None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:58.051129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc5e87-0161-4883-bd88-78f6f4c9679c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:58.051129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc5e87-0161-4883-bd88-78f6f4c9679c None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:58.052651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 729/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:03:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:03:58.059322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa281f4-87f9-4d16-8164-91c4926dec91 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:58.059950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa281f4-87f9-4d16-8164-91c4926dec91 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:58.059950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa281f4-87f9-4d16-8164-91c4926dec91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:58.129212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:58.152101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-afa281f4-87f9-4d16-8164-91c4926dec91 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:03:58.153378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-afa281f4-87f9-4d16-8164-91c4926dec91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:58.268762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 719/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:03:58.312398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc5e87-0161-4883-bd88-78f6f4c9679c None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:03:58.313978 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b1fc5e87-0161-4883-bd88-78f6f4c9679c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:58.439849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc5e87-0161-4883-bd88-78f6f4c9679c None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:03:58.484050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:03:58.521015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:03:58.521540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:03:58.521834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:03:58.522104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:03:58.523741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:03:58.646330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:03:58.647917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ae5ad5-4bd9-45bc-a83c-80427863aca0 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:03:58.667780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 730/1449] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:03:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:00.986698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34a3e4-3120-49af-ae5d-d63b8614dcb2 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:00.987304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34a3e4-3120-49af-ae5d-d63b8614dcb2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:00.987304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e34a3e4-3120-49af-ae5d-d63b8614dcb2 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:00.988647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 720/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:00.994225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3d4e70-44bf-4486-a94f-1d859a35b6e1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:00.994712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3d4e70-44bf-4486-a94f-1d859a35b6e1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:00.994828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3d4e70-44bf-4486-a94f-1d859a35b6e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:01.063916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8c3d4e70-44bf-4486-a94f-1d859a35b6e1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:01.065212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8c3d4e70-44bf-4486-a94f-1d859a35b6e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:01.086180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:01.165803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 731/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:01.208078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34a3e4-3120-49af-ae5d-d63b8614dcb2 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:01.209682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2e34a3e4-3120-49af-ae5d-d63b8614dcb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:01.319496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e34a3e4-3120-49af-ae5d-d63b8614dcb2 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:01.340505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:01.395096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:01.395701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:01.395975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:01.396264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:01.397861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:01.510092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:01.511468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341b1704-6921-4668-8443-0735edf08618 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:01.532969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 721/1452] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:03.854695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305bfa86-900d-46e1-bb8e-cfe306887219 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:03.855228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305bfa86-900d-46e1-bb8e-cfe306887219 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:03.855228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305bfa86-900d-46e1-bb8e-cfe306887219 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:03.856538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 732/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:03.861993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79037fb0-bc60-4811-85c7-a65e1562e07d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:03.862474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79037fb0-bc60-4811-85c7-a65e1562e07d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:03.862474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79037fb0-bc60-4811-85c7-a65e1562e07d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:03.938169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:03.941443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-79037fb0-bc60-4811-85c7-a65e1562e07d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:03.943267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-79037fb0-bc60-4811-85c7-a65e1562e07d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:04.047911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 722/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:04.087014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305bfa86-900d-46e1-bb8e-cfe306887219 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:04.088609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-305bfa86-900d-46e1-bb8e-cfe306887219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:04.198327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305bfa86-900d-46e1-bb8e-cfe306887219 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:04.250588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:04.284227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:04.284869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:04.285151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:04.285433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:04.287056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:04.398571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:04.400018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b569849-06a5-4398-a98c-4f3a031a95e3 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:04.420173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 733/1455] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:06.550130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e50cb9-1070-476d-99e8-2c661afe6e41 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:06.550665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e50cb9-1070-476d-99e8-2c661afe6e41 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:06.550665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e50cb9-1070-476d-99e8-2c661afe6e41 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:06.551945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 723/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:06.557562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bcd19-3a83-49c0-99c1-a1574ca1c0b5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:06.557994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bcd19-3a83-49c0-99c1-a1574ca1c0b5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:06.558186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bcd19-3a83-49c0-99c1-a1574ca1c0b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:06.627255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9d3bcd19-3a83-49c0-99c1-a1574ca1c0b5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:06.628564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9d3bcd19-3a83-49c0-99c1-a1574ca1c0b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:06.662522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:06.738829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 734/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:06.779441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e50cb9-1070-476d-99e8-2c661afe6e41 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:06.781066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-48e50cb9-1070-476d-99e8-2c661afe6e41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:06.890533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e50cb9-1070-476d-99e8-2c661afe6e41 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:06.919855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:06.973786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:06.974392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:06.974670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:06.974942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:06.976656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:07.092606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:07.094050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f529dba-3ecc-4763-b420-cb2d9b02ebe5 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:07.112905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 724/1458] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:09.256826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa4edd8-c923-4ffb-8ee5-89b8f0ec2148 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:09.257346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa4edd8-c923-4ffb-8ee5-89b8f0ec2148 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:09.257346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa4edd8-c923-4ffb-8ee5-89b8f0ec2148 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:09.258576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 735/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:04:09.264638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a45dd-e02d-46c1-8a6d-c0abb9c3d194 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:09.265169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a45dd-e02d-46c1-8a6d-c0abb9c3d194 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:09.265169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125a45dd-e02d-46c1-8a6d-c0abb9c3d194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:09.339020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-125a45dd-e02d-46c1-8a6d-c0abb9c3d194 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:09.340360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-125a45dd-e02d-46c1-8a6d-c0abb9c3d194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:09.351588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:09.459617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 725/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:09.498350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa4edd8-c923-4ffb-8ee5-89b8f0ec2148 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:09.499909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1fa4edd8-c923-4ffb-8ee5-89b8f0ec2148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:09.616838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa4edd8-c923-4ffb-8ee5-89b8f0ec2148 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:09.680465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:09.701786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:09.702422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:09.702701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:09.702972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:09.704644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:09.816211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:09.817676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf804d3-792c-4064-b6d4-8abffb5e6691 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:09.837404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 736/1461] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:04:11.937478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48aee6c-b335-41c9-822f-24a3af4c8b4d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:11.938091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48aee6c-b335-41c9-822f-24a3af4c8b4d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:11.938091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48aee6c-b335-41c9-822f-24a3af4c8b4d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:11.939266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 726/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:11.944963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81cfc5e-590b-45e3-b345-b2cfb6f93b65 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:11.945453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81cfc5e-590b-45e3-b345-b2cfb6f93b65 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:11.945583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81cfc5e-590b-45e3-b345-b2cfb6f93b65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:12.027719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c81cfc5e-590b-45e3-b345-b2cfb6f93b65 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:12.030692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c81cfc5e-590b-45e3-b345-b2cfb6f93b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:12.071462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:12.134150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 737/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:12.174311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48aee6c-b335-41c9-822f-24a3af4c8b4d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:12.175997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d48aee6c-b335-41c9-822f-24a3af4c8b4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:12.290328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48aee6c-b335-41c9-822f-24a3af4c8b4d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:12.329497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:12.366493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:12.367097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:12.367376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:12.367647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:12.369283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:12.483649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:12.485120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d5e5e4-54b2-474a-a983-df84449343d6 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:12.504282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 727/1464] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:14.636616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05771fbc-5f09-404a-9dea-2235c03bec17 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:14.637131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05771fbc-5f09-404a-9dea-2235c03bec17 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:14.637131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05771fbc-5f09-404a-9dea-2235c03bec17 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:14.638466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 738/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:14.644020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676a18b-623f-4d5e-a13b-59129c446bca None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:14.644510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676a18b-623f-4d5e-a13b-59129c446bca None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:14.644617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7676a18b-623f-4d5e-a13b-59129c446bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:14.711840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7676a18b-623f-4d5e-a13b-59129c446bca None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:14.713077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7676a18b-623f-4d5e-a13b-59129c446bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:14.731946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:14.808953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 728/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:14.847083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05771fbc-5f09-404a-9dea-2235c03bec17 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:14.848688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-05771fbc-5f09-404a-9dea-2235c03bec17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:14.957205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05771fbc-5f09-404a-9dea-2235c03bec17 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:15.026349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:15.034015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:15.034626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:15.034914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:15.035169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:15.036805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:15.146434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:15.147864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9dbefb-59fd-4a37-933e-9a8136a5c4b1 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:15.166907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 739/1467] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:17.267093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04291d59-3ac0-4c52-9eeb-f125982907ab None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:17.267672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04291d59-3ac0-4c52-9eeb-f125982907ab None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:17.267672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04291d59-3ac0-4c52-9eeb-f125982907ab None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:17.269020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 729/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:04:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:17.274535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d371e4c-1b22-43c1-875d-89cc1c6921a6 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:17.274999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d371e4c-1b22-43c1-875d-89cc1c6921a6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:17.275112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d371e4c-1b22-43c1-875d-89cc1c6921a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:17.347520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8d371e4c-1b22-43c1-875d-89cc1c6921a6 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:17.348800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8d371e4c-1b22-43c1-875d-89cc1c6921a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:17.399764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:17.451797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 740/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:17.490960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04291d59-3ac0-4c52-9eeb-f125982907ab None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:17.492591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-04291d59-3ac0-4c52-9eeb-f125982907ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:17.602561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04291d59-3ac0-4c52-9eeb-f125982907ab None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:17.651062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:17.678065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:17.678692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:17.678976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:17.679253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:17.680985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:17.796310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:17.797766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01dae492-2221-4b21-be02-c288be2baefe service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:17.817051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 730/1470] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:27.449977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=22 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:27.814695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=66 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:38.142315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6472c6-3173-43c8-9e92-d506c5e1cce2 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:38.142962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6472c6-3173-43c8-9e92-d506c5e1cce2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:38.142962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6472c6-3173-43c8-9e92-d506c5e1cce2 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:38.144446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 741/1471] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:38.151476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9026969-117c-47c9-886b-4552a1663e31 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:38.152175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9026969-117c-47c9-886b-4552a1663e31 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:38.152354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9026969-117c-47c9-886b-4552a1663e31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:38.251993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f9026969-117c-47c9-886b-4552a1663e31 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:38.253620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f9026969-117c-47c9-886b-4552a1663e31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:38.383098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 731/1472] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:38.477617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4f585-c1c2-43f2-82f7-a70c609aac90 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:38.477902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4f585-c1c2-43f2-82f7-a70c609aac90 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:38.478179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4f585-c1c2-43f2-82f7-a70c609aac90 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:38.479572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 742/1473] 10.222.0.26 () {54 vars in 966 bytes} [Sun Aug 7 01:04:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:38.484976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e058d84-89da-4333-8a74-1dcfb636afc9 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:38.485265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e058d84-89da-4333-8a74-1dcfb636afc9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:38.485548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e058d84-89da-4333-8a74-1dcfb636afc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:38.572526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4e058d84-89da-4333-8a74-1dcfb636afc9 None None] MFA Rules not processed for user `7bd8b2ff3ce14290ac0cf476021e3496`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:38.573772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4e058d84-89da-4333-8a74-1dcfb636afc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:38.596105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:38.654077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 732/1474] 10.222.0.26 () {58 vars in 1063 bytes} [Sun Aug 7 01:04:38 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:38.661692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c4f585-c1c2-43f2-82f7-a70c609aac90 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:38.663220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-19c4f585-c1c2-43f2-82f7-a70c609aac90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:38.777036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c4f585-c1c2-43f2-82f7-a70c609aac90 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:38.861531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:38.862056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:38.862334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:38.862609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:38.864308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:38.923877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:38.986308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:38.987795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cba07e-5de2-4754-9047-0d9411bdaa5f service glance] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:39.011050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 743/1475] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:41.263470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10852058-9731-4089-8912-30db67143870 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:41.264116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10852058-9731-4089-8912-30db67143870 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:41.264116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10852058-9731-4089-8912-30db67143870 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:41.265487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 733/1476] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:41.272299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383b0c47-9854-4b7a-8771-ab0bb1f24181 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:41.272994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383b0c47-9854-4b7a-8771-ab0bb1f24181 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:41.272994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383b0c47-9854-4b7a-8771-ab0bb1f24181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:41.359383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-383b0c47-9854-4b7a-8771-ab0bb1f24181 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:41.360666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-383b0c47-9854-4b7a-8771-ab0bb1f24181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:41.476983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 744/1477] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:04:41.501695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10852058-9731-4089-8912-30db67143870 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:41.503367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-10852058-9731-4089-8912-30db67143870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:41.668586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10852058-9731-4089-8912-30db67143870 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:41.731611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:41.780563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:41.781111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:41.781380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:41.781646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:41.782285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:41.784066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:04:41.784066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:04:41.784066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:04:41.785247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048cb664-88ce-4b7c-b363-995722654f03 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:41.806458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 734/1478] 10.222.0.26 () {56 vars in 1108 bytes} [Sun Aug 7 01:04:41 2022] GET /identity/v3/projects => generated 2553 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:04:44.019043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e068c3-d0ae-4f86-af3e-46ac33359713 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:44.019588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e068c3-d0ae-4f86-af3e-46ac33359713 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:44.019588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e068c3-d0ae-4f86-af3e-46ac33359713 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:44.020983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 745/1479] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:44.026794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5dd96e-7cbb-4a23-ab6a-0c26bfff1142 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:44.027305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5dd96e-7cbb-4a23-ab6a-0c26bfff1142 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:44.027404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5dd96e-7cbb-4a23-ab6a-0c26bfff1142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:44.095827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4a5dd96e-7cbb-4a23-ab6a-0c26bfff1142 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:44.097094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4a5dd96e-7cbb-4a23-ab6a-0c26bfff1142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:44.196745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 735/1480] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:44.241299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e068c3-d0ae-4f86-af3e-46ac33359713 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:44.242860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d1e068c3-d0ae-4f86-af3e-46ac33359713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:44.253725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:44.398042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e068c3-d0ae-4f86-af3e-46ac33359713 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:44.477576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:44.510822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:44.512035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:44.512321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:44.512597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:44.514211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:44.609202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:44.610748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8669ea-0a58-4c64-a3ab-1008e5205b75 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:44.631165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 746/1481] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:46.918769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4644c-2f6c-4cc7-8783-650c0ed3b826 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:46.919388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4644c-2f6c-4cc7-8783-650c0ed3b826 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:46.919388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4644c-2f6c-4cc7-8783-650c0ed3b826 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:46.920749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 736/1482] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:46.926730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb42c2aa-53f1-4c02-8525-5c37c6f3c579 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:46.927220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb42c2aa-53f1-4c02-8525-5c37c6f3c579 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:46.927327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb42c2aa-53f1-4c02-8525-5c37c6f3c579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:47.009487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-eb42c2aa-53f1-4c02-8525-5c37c6f3c579 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:47.011929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-eb42c2aa-53f1-4c02-8525-5c37c6f3c579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:47.069744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:47.139746 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 747/1483] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:47.181056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4644c-2f6c-4cc7-8783-650c0ed3b826 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:47.182669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-78e4644c-2f6c-4cc7-8783-650c0ed3b826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:47.207474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:47.292476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4644c-2f6c-4cc7-8783-650c0ed3b826 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:47.376404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:47.377017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:47.377328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:47.377608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:47.379230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:47.467178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:47.503335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:47.504929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59c704c-86a1-4ea7-b4f4-6a795066a10a service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:47.523701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 737/1484] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:49.643860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8827597-6ee7-4ccb-b48a-1f5c844fe252 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:49.644519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8827597-6ee7-4ccb-b48a-1f5c844fe252 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:49.644519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8827597-6ee7-4ccb-b48a-1f5c844fe252 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:49.645875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 748/1485] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:49.652520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3694adb-203a-4ab3-8a52-61ab8ea07190 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:49.653165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3694adb-203a-4ab3-8a52-61ab8ea07190 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:49.653165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3694adb-203a-4ab3-8a52-61ab8ea07190 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:49.722757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c3694adb-203a-4ab3-8a52-61ab8ea07190 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:49.723977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c3694adb-203a-4ab3-8a52-61ab8ea07190 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:49.821990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 738/1486] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:49.862021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8827597-6ee7-4ccb-b48a-1f5c844fe252 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:49.863575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d8827597-6ee7-4ccb-b48a-1f5c844fe252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:49.978351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8827597-6ee7-4ccb-b48a-1f5c844fe252 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:50.060975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:50.062879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:50.063530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:50.064007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:50.064325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:50.066635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:50.177423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:50.178774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb971c5-9250-4825-80d0-95d693fc9117 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:50.199696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 749/1487] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:52.373897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fab0d1-1e56-4df9-b56f-f0348ce582c9 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:52.374469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fab0d1-1e56-4df9-b56f-f0348ce582c9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:52.374469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fab0d1-1e56-4df9-b56f-f0348ce582c9 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:52.375774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 739/1488] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:52.381312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feedf98-b083-4a8a-87d3-8d9892845609 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:52.381754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feedf98-b083-4a8a-87d3-8d9892845609 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:52.381955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feedf98-b083-4a8a-87d3-8d9892845609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:52.449999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2feedf98-b083-4a8a-87d3-8d9892845609 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:52.451327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2feedf98-b083-4a8a-87d3-8d9892845609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:52.503611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:52.548116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 750/1489] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:52.590131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fab0d1-1e56-4df9-b56f-f0348ce582c9 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:52.591691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a6fab0d1-1e56-4df9-b56f-f0348ce582c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:52.632478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:52.709875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fab0d1-1e56-4df9-b56f-f0348ce582c9 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:52.791398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:52.792007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:52.792351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:52.792624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:52.794318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:52.887697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:52.918496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:52.919944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c3f4ee-0f8c-42b0-b9f5-aadc4c8ac860 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:52.941329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 740/1490] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:04:55.219792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead88035-7330-4201-879a-6edb2d689891 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:55.220386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead88035-7330-4201-879a-6edb2d689891 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:55.220489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead88035-7330-4201-879a-6edb2d689891 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:55.221821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 751/1491] 10.222.0.26 () {54 vars in 927 bytes} [Sun Aug 7 01:04:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:04:55.227350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a78ed6-c76c-4338-8c42-aacfa92cb899 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:55.227797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a78ed6-c76c-4338-8c42-aacfa92cb899 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:55.227937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a78ed6-c76c-4338-8c42-aacfa92cb899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:55.298364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d2a78ed6-c76c-4338-8c42-aacfa92cb899 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:04:55.299598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d2a78ed6-c76c-4338-8c42-aacfa92cb899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:55.400850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 741/1492] 10.222.0.26 () {58 vars in 1024 bytes} [Sun Aug 7 01:04:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:04:55.453188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead88035-7330-4201-879a-6edb2d689891 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:04:55.454751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ead88035-7330-4201-879a-6edb2d689891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:55.558670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead88035-7330-4201-879a-6edb2d689891 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:04:55.632154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:04:55.632789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:04:55.633064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:04:55.633330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:04:55.634913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:04:55.643991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:04:55.743231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:04:55.744707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f907b2c-90f7-4c67-a206-f7ab80e3d09e service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:04:55.762998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 752/1493] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:04:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:05:05.398957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=82 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:05:05.761669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=50 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:10.583011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:10.583593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:10.583593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:10.712071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:06:10.714153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:10.845739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 742/1494] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:06:10.854620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:10.855221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:10.855221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:10.938395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:06:10.939859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:11.068078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 753/1495] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:06:11.073999 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:11.076175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:11.096785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a87371-bae1-4618-9231-5ac3aa9342a4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:11.204247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:11.268819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:11.269980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:11.270536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:11.271146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:11.272503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:06:11.273098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:11.275815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:11.275815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:06:11.275815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:06:11.277579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:11.299139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 743/1496] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:06:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:06:11.307324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:11.317206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e61625-2d59-4ce3-9fed-ee7cc3d022e0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:11.386332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:11.422205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:11.422813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:11.423095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:11.423372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:11.423998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:11.425767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:11.425767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:06:11.425767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:06:11.426974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:11.489446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 754/1497] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:06:11 2022] POST /identity/v3/projects => generated 379 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:06:11.495089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:11.496895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:11.532115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f92ee-f4a7-4d6f-a2ca-eccd3721d464 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:11.649673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:11.650463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:11.651047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:11.651047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:11.651381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:11.652442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:11.652442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:06:11.652442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:06:11.653670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:11.702144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 744/1498] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:06:11 2022] POST /identity/v3/users => generated 290 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:06:11.706931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:11.708580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:11.777789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50212584-0892-473d-ba0f-dc4178205ae1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:11.889089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:11.912597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:11.913525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:11.914009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:11.914480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:11.916201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:11.917594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:11.917594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:06:11.917594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:06:11.919096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:11.943118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 755/1499] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:06:11 2022] GET /identity/v3/roles => generated 1420 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:06:11.947992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:11.949697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:11.962319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b76973-9599-4395-8294-a16df323a88c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:11.992532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:12.081860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:12.082474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:12.082757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:12.083040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:12.083685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:12.084736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:12.084736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:06:12.084736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:06:12.085622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9da332b-2bf8-497f-934f-6c441c618115 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:12.099823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 745/1500] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:06:11 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:06:12.104583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:12.106193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:12.142827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d884fbe-9c95-4634-8185-c162910721a7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:12.242114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:12.242686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:12.243465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:12.243736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] PATH_INFO: `/v3/projects/b704fca56aab4b67ade09b375d1934ec/users/79fb1c19339e4fab910edc328216e3d7/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:12.266798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b704fca56aab4b67ade09b375d1934ec, user_id=79fb1c19339e4fab910edc328216e3d7, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:12.267850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:12.267850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:06:12.267850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:06:12.269841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d5731e-51ed-4d0f-8477-ba288bf5080b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:12.302323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 756/1501] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:06:12 2022] PUT /identity/v3/projects/b704fca56aab4b67ade09b375d1934ec/users/79fb1c19339e4fab910edc328216e3d7/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:06:12.316461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127161d-19f5-471c-ab93-0b4ba5dcab34 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:12.316821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127161d-19f5-471c-ab93-0b4ba5dcab34 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:12.317168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5127161d-19f5-471c-ab93-0b4ba5dcab34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:12.378717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5127161d-19f5-471c-ab93-0b4ba5dcab34 None None] MFA Rules not processed for user `79fb1c19339e4fab910edc328216e3d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:06:12.380344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5127161d-19f5-471c-ab93-0b4ba5dcab34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:12.475599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 746/1502] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:06:12.491590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9cbce-1b3b-4b00-b619-edafbf673259 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:12.491878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9cbce-1b3b-4b00-b619-edafbf673259 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:12.492296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9cbce-1b3b-4b00-b619-edafbf673259 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:12.494309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 757/1503] 10.222.0.26 () {54 vars in 964 bytes} [Sun Aug 7 01:06:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:06:12.500522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd11fe-7b72-4d81-a313-ae279ea86985 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:12.500813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd11fe-7b72-4d81-a313-ae279ea86985 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:12.501095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bd11fe-7b72-4d81-a313-ae279ea86985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:12.537123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:12.590030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b9bd11fe-7b72-4d81-a313-ae279ea86985 None None] MFA Rules not processed for user `f2a3c404c5f840368281af8532b7b46b`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:06:12.591284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b9bd11fe-7b72-4d81-a313-ae279ea86985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:12.674972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 747/1504] 10.222.0.26 () {58 vars in 1061 bytes} [Sun Aug 7 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:06:12.684262 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a9cbce-1b3b-4b00-b619-edafbf673259 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:12.685806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-60a9cbce-1b3b-4b00-b619-edafbf673259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:12.738988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:12.800717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a9cbce-1b3b-4b00-b619-edafbf673259 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:12.881988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:12.882600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:12.882879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:12.883148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:12.884877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:13.008185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:13.010323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:13.014535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:13.032952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 758/1505] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:06:12 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:06:13.081179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc30dc38-de8d-4dd3-b2bb-564c2c55e9b9 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:13.081475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc30dc38-de8d-4dd3-b2bb-564c2c55e9b9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:13.081751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc30dc38-de8d-4dd3-b2bb-564c2c55e9b9 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:13.083054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 748/1506] 10.222.0.26 () {52 vars in 830 bytes} [Sun Aug 7 01:06:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:06:13.087449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:13.088817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:13.199754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee6e0e8-f736-4437-9061-489043e1a223 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:13.336029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:13.337459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:13.337672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:13.337953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] PATH_INFO: `/v3/users/79fb1c19339e4fab910edc328216e3d7` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:13.353023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79fb1c19339e4fab910edc328216e3d7)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:13.354193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:13.354193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:06:13.354193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:06:13.355680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:13.367172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:13.432933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79fb1c19339e4fab910edc328216e3d7'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:13.443109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79fb1c19339e4fab910edc328216e3d7'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:13.458959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79fb1c19339e4fab910edc328216e3d7'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:13.459553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79fb1c19339e4fab910edc328216e3d7'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:13.459970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79fb1c19339e4fab910edc328216e3d7'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:13.493513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 759/1507] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:06:13 2022] DELETE /identity/v3/users/79fb1c19339e4fab910edc328216e3d7 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:06:13.498249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:13.498583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:13.498903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:13.566123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:06:13.567370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:13.652779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:13.668101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 749/1508] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:06:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:06:13.678135 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:13.679429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:13.798412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e194cd3-a6d5-4f07-a09a-b27942a26574 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:13.858602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:13.882281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:13.882910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:13.883211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:13.883508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:13.885366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:14.019394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:14.021191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1db040d-bc6a-4e82-8168-cbcef3ce81dd service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:14.043867 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 760/1509] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:06:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:06:14.428817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:06:14.430230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:06:14.453706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377eefc7-6fc4-4bae-a10e-bb238fa0a241 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:06:14.601512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:06:14.602404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:06:14.602789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:06:14.603163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] PATH_INFO: `/v3/projects/b704fca56aab4b67ade09b375d1934ec` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:06:14.616687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b704fca56aab4b67ade09b375d1934ec)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:06:14.618539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:06:14.618539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:06:14.618539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:06:14.620172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:06:14.682147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b704fca56aab4b67ade09b375d1934ec', 'request_id': 'req-3e9c5186-bda2-467b-af34-b29435a4f218'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:14.688651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b704fca56aab4b67ade09b375d1934ec', 'request_id': 'req-3e9c5186-bda2-467b-af34-b29435a4f218'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:14.698311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b704fca56aab4b67ade09b375d1934ec', 'request_id': 'req-3e9c5186-bda2-467b-af34-b29435a4f218'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:14.731992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] The token cache is being invalidate because project b704fca56aab4b67ade09b375d1934ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:06:14.732318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:14.732917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3e9c5186-bda2-467b-af34-b29435a4f218 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:06:14.734277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 750/1510] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:06:14 2022] DELETE /identity/v3/projects/b704fca56aab4b67ade09b375d1934ec => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:06:24.041395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=62 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:06:24.727710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:07:46.333855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be65273-a540-4cef-bcc6-979ea677ca05 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:46.334477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be65273-a540-4cef-bcc6-979ea677ca05 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:46.334477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be65273-a540-4cef-bcc6-979ea677ca05 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:46.335846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 761/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:07:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:07:46.341546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc7b0ed-3007-499c-9ec9-ec827ca0c108 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:46.342128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc7b0ed-3007-499c-9ec9-ec827ca0c108 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:46.342128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc7b0ed-3007-499c-9ec9-ec827ca0c108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:46.441072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3cc7b0ed-3007-499c-9ec9-ec827ca0c108 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:07:46.442348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3cc7b0ed-3007-499c-9ec9-ec827ca0c108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:07:46.565469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 751/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:07:46.728631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be65273-a540-4cef-bcc6-979ea677ca05 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:07:46.731180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8be65273-a540-4cef-bcc6-979ea677ca05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:07:46.878331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be65273-a540-4cef-bcc6-979ea677ca05 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:07:46.885387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cac8343-270b-43fd-959a-c2ddd4627a1e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:07:46.886094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac8343-270b-43fd-959a-c2ddd4627a1e service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:46.886528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac8343-270b-43fd-959a-c2ddd4627a1e service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:46.886888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac8343-270b-43fd-959a-c2ddd4627a1e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:46.896599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cac8343-270b-43fd-959a-c2ddd4627a1e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:07:46.898873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cac8343-270b-43fd-959a-c2ddd4627a1e service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:07:46.910088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 762/1513] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:07:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:07:47.805507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea16981-4610-4aa7-a066-bf8b564478df None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:47.806096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea16981-4610-4aa7-a066-bf8b564478df None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:47.806096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea16981-4610-4aa7-a066-bf8b564478df None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:47.807583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 752/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:07:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:07:47.813583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c002496e-713f-4cda-835e-700bc83b2ce4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:47.814152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c002496e-713f-4cda-835e-700bc83b2ce4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:47.814152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c002496e-713f-4cda-835e-700bc83b2ce4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:47.837225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c002496e-713f-4cda-835e-700bc83b2ce4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:07:47.839207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c002496e-713f-4cda-835e-700bc83b2ce4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:07:47.849394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 763/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:07:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:07:47.921326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c6a90-4b37-4c10-a111-ec1bf068b7ce None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:47.922329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c6a90-4b37-4c10-a111-ec1bf068b7ce None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:47.922329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c6a90-4b37-4c10-a111-ec1bf068b7ce None None] PATH_INFO: `/v3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:47.924966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 753/1516] 10.222.0.17 () {54 vars in 925 bytes} [Sun Aug 7 01:07:47 2022] GET /identity/v3 => generated 255 bytes in 4 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:07:47.931632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a19e3b8-4fa0-4586-baf2-ab152daa0d2e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:47.932107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a19e3b8-4fa0-4586-baf2-ab152daa0d2e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:47.932443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a19e3b8-4fa0-4586-baf2-ab152daa0d2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:48.011377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3a19e3b8-4fa0-4586-baf2-ab152daa0d2e None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:07:48.012786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3a19e3b8-4fa0-4586-baf2-ab152daa0d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:07:48.017268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c6a90-4b37-4c10-a111-ec1bf068b7ce None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:07:48.043242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c6a90-4b37-4c10-a111-ec1bf068b7ce None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:07:48.049876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d358a1-eccc-4c5d-a28a-ae4d7df0ad6b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:07:48.050507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d358a1-eccc-4c5d-a28a-ae4d7df0ad6b service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:48.050778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d358a1-eccc-4c5d-a28a-ae4d7df0ad6b service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:48.051053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d358a1-eccc-4c5d-a28a-ae4d7df0ad6b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:48.062108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d358a1-eccc-4c5d-a28a-ae4d7df0ad6b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:07:48.064147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d358a1-eccc-4c5d-a28a-ae4d7df0ad6b service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:07:48.076467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 754/1517] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:07:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:07:48.114604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:07:48.146771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 764/1518] 10.222.0.17 () {58 vars in 1015 bytes} [Sun Aug 7 01:07:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:07:52.125004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1b407-222a-4def-b701-4fdff3127924 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:52.125789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1b407-222a-4def-b701-4fdff3127924 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:52.125789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1b407-222a-4def-b701-4fdff3127924 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:52.127931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 755/1519] 10.222.0.26 () {54 vars in 961 bytes} [Sun Aug 7 01:07:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:07:52.135271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121f26e-e691-49a4-87a0-3afb7d4d81bb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:52.135844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121f26e-e691-49a4-87a0-3afb7d4d81bb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:52.136344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121f26e-e691-49a4-87a0-3afb7d4d81bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:52.157845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-f121f26e-e691-49a4-87a0-3afb7d4d81bb None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:07:52.159086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f121f26e-e691-49a4-87a0-3afb7d4d81bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:07:52.168772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 765/1520] 10.222.0.26 () {58 vars in 1058 bytes} [Sun Aug 7 01:07:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:07:52.176575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1b407-222a-4def-b701-4fdff3127924 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:07:52.194329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf1b407-222a-4def-b701-4fdff3127924 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:07:52.199122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304517fc-d7de-4c58-81c5-746e7b2a645e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:07:52.200152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304517fc-d7de-4c58-81c5-746e7b2a645e service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:52.200682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304517fc-d7de-4c58-81c5-746e7b2a645e service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:52.201160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304517fc-d7de-4c58-81c5-746e7b2a645e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:52.212540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304517fc-d7de-4c58-81c5-746e7b2a645e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:07:52.214672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304517fc-d7de-4c58-81c5-746e7b2a645e service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:07:52.226611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 756/1521] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:07:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:07:59.432492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805d64b1-148f-4069-8988-fb0fd21737e6 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:59.433065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805d64b1-148f-4069-8988-fb0fd21737e6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:59.433065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805d64b1-148f-4069-8988-fb0fd21737e6 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:59.434419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 766/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:07:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:07:59.439907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695f8ebd-dccc-4e22-a888-51286058bcab None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:59.440590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695f8ebd-dccc-4e22-a888-51286058bcab None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:59.440590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695f8ebd-dccc-4e22-a888-51286058bcab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:59.461466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-695f8ebd-dccc-4e22-a888-51286058bcab None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:07:59.462697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-695f8ebd-dccc-4e22-a888-51286058bcab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:07:59.474777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 757/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:07:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:07:59.629388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805d64b1-148f-4069-8988-fb0fd21737e6 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:07:59.645322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805d64b1-148f-4069-8988-fb0fd21737e6 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:07:59.651356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d997b52-13ae-457f-8001-5d392b5bf7db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:07:59.651973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d997b52-13ae-457f-8001-5d392b5bf7db service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:07:59.652284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d997b52-13ae-457f-8001-5d392b5bf7db service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:07:59.652570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d997b52-13ae-457f-8001-5d392b5bf7db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:07:59.662229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d997b52-13ae-457f-8001-5d392b5bf7db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:07:59.664004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d997b52-13ae-457f-8001-5d392b5bf7db service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:07:59.675837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 767/1524] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:07:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:08:00.590341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d0d8f4-126c-4d1d-b1c2-f2000c91114c None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:00.590908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d0d8f4-126c-4d1d-b1c2-f2000c91114c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:00.590908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d0d8f4-126c-4d1d-b1c2-f2000c91114c None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:00.592567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 758/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 7 01:08:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:08:00.598169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3834fb-1b79-4569-a9b2-a9132ed391ff None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:00.598690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3834fb-1b79-4569-a9b2-a9132ed391ff None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:00.598824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3834fb-1b79-4569-a9b2-a9132ed391ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:00.622397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9d3834fb-1b79-4569-a9b2-a9132ed391ff None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:08:00.623889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9d3834fb-1b79-4569-a9b2-a9132ed391ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:00.634026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 768/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 7 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:08:00.797979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d0d8f4-126c-4d1d-b1c2-f2000c91114c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:08:00.813949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d0d8f4-126c-4d1d-b1c2-f2000c91114c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:08:00.820265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ced4955-d2b9-46de-aa19-0a10d18e0878 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:08:00.820913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ced4955-d2b9-46de-aa19-0a10d18e0878 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:00.821195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ced4955-d2b9-46de-aa19-0a10d18e0878 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:00.821467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ced4955-d2b9-46de-aa19-0a10d18e0878 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:00.830881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ced4955-d2b9-46de-aa19-0a10d18e0878 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:08:00.832782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ced4955-d2b9-46de-aa19-0a10d18e0878 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:08:00.843828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 759/1527] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:08:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:08:04.910653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f94b02-f32c-45d2-a66c-89f9a04dc020 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:04.911400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f94b02-f32c-45d2-a66c-89f9a04dc020 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:04.911400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f94b02-f32c-45d2-a66c-89f9a04dc020 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:04.913372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 769/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:08:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:08:04.919428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c81f5-3eac-48e9-b1cc-e1401c87d07a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:04.919945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c81f5-3eac-48e9-b1cc-e1401c87d07a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:04.920393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c81f5-3eac-48e9-b1cc-e1401c87d07a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:04.949083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ff6c81f5-3eac-48e9-b1cc-e1401c87d07a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:08:04.950327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ff6c81f5-3eac-48e9-b1cc-e1401c87d07a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:04.959094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 760/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:08:05.041915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f94b02-f32c-45d2-a66c-89f9a04dc020 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:08:05.043684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-34f94b02-f32c-45d2-a66c-89f9a04dc020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:05.129902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f94b02-f32c-45d2-a66c-89f9a04dc020 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:08:05.133715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4d1465-b75e-44ff-b5a4-ffd983c7ae81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:08:05.134336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4d1465-b75e-44ff-b5a4-ffd983c7ae81 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:05.134619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4d1465-b75e-44ff-b5a4-ffd983c7ae81 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:05.134887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4d1465-b75e-44ff-b5a4-ffd983c7ae81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:05.145101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4d1465-b75e-44ff-b5a4-ffd983c7ae81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:08:05.146709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4d1465-b75e-44ff-b5a4-ffd983c7ae81 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:08:05.158289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 770/1530] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:08:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:08:07.213285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eea190-2522-4f7c-aa30-0c3e9de5ef42 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:07.213837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eea190-2522-4f7c-aa30-0c3e9de5ef42 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:07.213837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eea190-2522-4f7c-aa30-0c3e9de5ef42 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:07.215185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 761/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:08:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:08:07.220865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd964a3-86e7-4c79-8f21-193738eb476b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:07.221353 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd964a3-86e7-4c79-8f21-193738eb476b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:07.221454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd964a3-86e7-4c79-8f21-193738eb476b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:07.242346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9fd964a3-86e7-4c79-8f21-193738eb476b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:08:07.243650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9fd964a3-86e7-4c79-8f21-193738eb476b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:07.251681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 771/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:08:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:08:07.289296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eea190-2522-4f7c-aa30-0c3e9de5ef42 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:08:07.291192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b0eea190-2522-4f7c-aa30-0c3e9de5ef42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:07.305971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eea190-2522-4f7c-aa30-0c3e9de5ef42 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:08:07.309670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c62b187-63bc-4c3b-aad3-7363fe96de7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:08:07.310282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c62b187-63bc-4c3b-aad3-7363fe96de7f service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:07.310558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c62b187-63bc-4c3b-aad3-7363fe96de7f service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:07.310828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c62b187-63bc-4c3b-aad3-7363fe96de7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:07.320991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c62b187-63bc-4c3b-aad3-7363fe96de7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:08:07.322431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c62b187-63bc-4c3b-aad3-7363fe96de7f service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:08:07.332463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 762/1533] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:08:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:08:11.220117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f1b65-1b81-4e91-9865-cc43d95a0e87 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:11.220774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f1b65-1b81-4e91-9865-cc43d95a0e87 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:11.220774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6f1b65-1b81-4e91-9865-cc43d95a0e87 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:11.222501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 772/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:08:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:08:11.230611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126845d-936c-44a3-a51a-4cc78e671dda None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:11.231227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126845d-936c-44a3-a51a-4cc78e671dda None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:11.231227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126845d-936c-44a3-a51a-4cc78e671dda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:11.239352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:08:11.254591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3126845d-936c-44a3-a51a-4cc78e671dda None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:08:11.255843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3126845d-936c-44a3-a51a-4cc78e671dda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:11.265478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 763/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:08:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:08:11.587155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f1b65-1b81-4e91-9865-cc43d95a0e87 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:08:11.590142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8b6f1b65-1b81-4e91-9865-cc43d95a0e87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:11.702485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6f1b65-1b81-4e91-9865-cc43d95a0e87 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:08:11.706121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1298ebde-7093-406e-bad8-a598beef166f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:08:11.706751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1298ebde-7093-406e-bad8-a598beef166f service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:11.707036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1298ebde-7093-406e-bad8-a598beef166f service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:11.707313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1298ebde-7093-406e-bad8-a598beef166f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:11.718841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1298ebde-7093-406e-bad8-a598beef166f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:08:11.720351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1298ebde-7093-406e-bad8-a598beef166f service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:08:11.730996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 773/1536] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:08:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:08:13.902417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5540cb9-b446-409f-8ea7-d1085840f7e9 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:13.903067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5540cb9-b446-409f-8ea7-d1085840f7e9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:13.903067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5540cb9-b446-409f-8ea7-d1085840f7e9 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:13.904359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 764/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 7 01:08:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:08:13.910275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41068560-5add-49ce-a077-14b9e7809ef8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:13.910886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41068560-5add-49ce-a077-14b9e7809ef8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:13.910886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41068560-5add-49ce-a077-14b9e7809ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:13.934048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-41068560-5add-49ce-a077-14b9e7809ef8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:08:13.935291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-41068560-5add-49ce-a077-14b9e7809ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:13.943865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 774/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 7 01:08:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:08:14.465546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5540cb9-b446-409f-8ea7-d1085840f7e9 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:08:14.468054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f5540cb9-b446-409f-8ea7-d1085840f7e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:08:14.483063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5540cb9-b446-409f-8ea7-d1085840f7e9 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:08:14.486865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8def7ee-4d71-4207-9fb0-eee44e341e46 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:08:14.487479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8def7ee-4d71-4207-9fb0-eee44e341e46 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:08:14.487756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8def7ee-4d71-4207-9fb0-eee44e341e46 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:08:14.488028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8def7ee-4d71-4207-9fb0-eee44e341e46 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:08:14.498990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8def7ee-4d71-4207-9fb0-eee44e341e46 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:08:14.500441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8def7ee-4d71-4207-9fb0-eee44e341e46 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:08:14.511500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 765/1539] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:08:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:08:23.928288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:08:24.506189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=8 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:47.573238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f38cb07-3857-480d-b845-62bd32290f24 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.575729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f38cb07-3857-480d-b845-62bd32290f24 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.575729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f38cb07-3857-480d-b845-62bd32290f24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.631201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7f38cb07-3857-480d-b845-62bd32290f24 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:47.633394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f38cb07-3857-480d-b845-62bd32290f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:47.653879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 775/1540] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:47.657321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5649aaf-76a1-4695-894f-5294ee359890 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.657953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5649aaf-76a1-4695-894f-5294ee359890 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.657953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5649aaf-76a1-4695-894f-5294ee359890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.665775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.666301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.667019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.698185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c5649aaf-76a1-4695-894f-5294ee359890 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:47.699493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c5649aaf-76a1-4695-894f-5294ee359890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:47.717619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 766/1541] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:09:47.719709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:47.722032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:47.729355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.729888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.730025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.745259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 776/1542] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:47.751110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:47.774395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d13ad-ef74-43e2-a413-a0dec3463ad5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:47.781209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:47.788614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:47.798163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:47.798824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.799306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.799662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.800469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:47.800817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:47.802030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:09:47.802030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:09:47.802030 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:09:47.803309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:47.811313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 767/1543] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:09:47.820611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:47.841430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 777/1544] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:09:47.851906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:47.853835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3ee957-7a10-4a50-a16f-a4eeb8d008f7 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:47.864690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:47.865757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.866201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.866655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.876763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:47.877993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:47.881643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:47.885662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfbe531-5449-4d50-9276-683ca7fe9dd7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:47.902942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:47.903916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.904397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.904860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.905996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:47.918351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:47.924509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 768/1545] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:09:47.937898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:47.969121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc34040b-e767-4505-a395-1632f476a20e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:47.983260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:47.989472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:47.990156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:47.993707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:47.993707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:47.994189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:09:47.994189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:09:47.994189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:09:48.001216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.067757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 778/1546] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:47 2022] POST /identity/v3/projects => generated 365 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:48.075278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.091048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 769/1547] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:47 2022] POST /identity/v3/projects => generated 383 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:48.098849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.309380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6cbe89-ed79-4481-854a-f11be44af7d4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.327868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41103b7-f1ee-479f-af62-db1a5e76261c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.328981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41103b7-f1ee-479f-af62-db1a5e76261c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.329477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41103b7-f1ee-479f-af62-db1a5e76261c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.329974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41103b7-f1ee-479f-af62-db1a5e76261c admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.334076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41103b7-f1ee-479f-af62-db1a5e76261c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.337746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41103b7-f1ee-479f-af62-db1a5e76261c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.368028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04939124-1591-43fd-8e6a-d178025ec749 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.389161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 770/1548] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/users => generated 292 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:09:48.390573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195ca9e1-2849-4584-844c-e57e912e118b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.391328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195ca9e1-2849-4584-844c-e57e912e118b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.391618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195ca9e1-2849-4584-844c-e57e912e118b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.399080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.400209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.400975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.401477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.402949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.413650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:09:48.413650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:09:48.413650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:09:48.416029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.422749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-195ca9e1-2849-4584-844c-e57e912e118b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.424334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-195ca9e1-2849-4584-844c-e57e912e118b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.446626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 771/1549] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.452273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4c2648-54d0-4b74-9b0e-815d0f586fa7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.452273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4c2648-54d0-4b74-9b0e-815d0f586fa7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.452273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4c2648-54d0-4b74-9b0e-815d0f586fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.472878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ff4c2648-54d0-4b74-9b0e-815d0f586fa7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.479889 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ff4c2648-54d0-4b74-9b0e-815d0f586fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.487301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 779/1550] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/users => generated 283 bytes in 414 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:09:48.490944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.497010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 772/1551] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.499126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0063743-edee-4bf8-b5d9-37afb50bdb70 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.499652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0063743-edee-4bf8-b5d9-37afb50bdb70 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.500129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0063743-edee-4bf8-b5d9-37afb50bdb70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.513817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d79b9a-7c63-485c-96af-3c83159e14bc admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.523136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e86d68-6f9f-4d4c-980c-10bdca818123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.524258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e86d68-6f9f-4d4c-980c-10bdca818123 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.524757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a0063743-edee-4bf8-b5d9-37afb50bdb70 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.524932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e86d68-6f9f-4d4c-980c-10bdca818123 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.525284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e86d68-6f9f-4d4c-980c-10bdca818123 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.526441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a0063743-edee-4bf8-b5d9-37afb50bdb70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.526756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e86d68-6f9f-4d4c-980c-10bdca818123 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.530496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e86d68-6f9f-4d4c-980c-10bdca818123 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.537256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 773/1552] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.538786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.539237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.539535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.566258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 780/1553] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:48.569939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b547bf-528d-4b32-9492-712e65b09bcf None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.570668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.571189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b547bf-528d-4b32-9492-712e65b09bcf None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.572034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b547bf-528d-4b32-9492-712e65b09bcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.572227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.583730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 774/1554] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.587741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.605909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-f9b547bf-528d-4b32-9492-712e65b09bcf None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.608307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f9b547bf-528d-4b32-9492-712e65b09bcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.614250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b11c05-dce8-402a-b15e-f3d4de7fa780 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.622836 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07499a55-8673-4a9b-96f4-a11e3ea529c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.630123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07499a55-8673-4a9b-96f4-a11e3ea529c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.630123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07499a55-8673-4a9b-96f4-a11e3ea529c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.630123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07499a55-8673-4a9b-96f4-a11e3ea529c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.630123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07499a55-8673-4a9b-96f4-a11e3ea529c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.630751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 781/1555] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.630751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7774fb-cf89-4420-a0a2-1633004077a0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.630751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7774fb-cf89-4420-a0a2-1633004077a0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.631503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07499a55-8673-4a9b-96f4-a11e3ea529c9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.631670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7774fb-cf89-4420-a0a2-1633004077a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.651980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 775/1556] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:48.654381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.654884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.655363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.663766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8d7774fb-cf89-4420-a0a2-1633004077a0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.666390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8d7774fb-cf89-4420-a0a2-1633004077a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.681916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 782/1557] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.683930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.684429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.684906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.694517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.697272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.716137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 776/1558] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.717777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.725467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.727616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.736812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b960c7-fd68-4c85-a070-661ecc959d24 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.744476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 783/1559] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.746189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4fc715-0727-482e-ae63-6a4788e1f300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.747256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4fc715-0727-482e-ae63-6a4788e1f300 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.747755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4fc715-0727-482e-ae63-6a4788e1f300 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.748261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4fc715-0727-482e-ae63-6a4788e1f300 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.748679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.749554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4fc715-0727-482e-ae63-6a4788e1f300 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.752970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4fc715-0727-482e-ae63-6a4788e1f300 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.763766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bd1cf-480e-46f7-859b-832c4bde0bee None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.769779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.771017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.771562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.771827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.772779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:48.773278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.775160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388e71a5-a370-48c4-8345-0645cb82e5ce admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.781141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 777/1560] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:48.783526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.784067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.784625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.793624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 784/1561] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:09:48.796023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.796494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.796971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.824720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.827169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.828874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.831019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.841946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 778/1562] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.843090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.850531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 785/1563] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.852264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.861528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7110826b-cb7f-4878-a962-7c4559ceb09d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.869381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfa98f3-bb79-4813-927f-cbd6ea54a280 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.870070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0921b55-7ab5-42a4-97a4-b8f7cd34fe65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.871135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0921b55-7ab5-42a4-97a4-b8f7cd34fe65 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.871619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0921b55-7ab5-42a4-97a4-b8f7cd34fe65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.872166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0921b55-7ab5-42a4-97a4-b8f7cd34fe65 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.873599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0921b55-7ab5-42a4-97a4-b8f7cd34fe65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.875657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.876495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.876828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.877033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0921b55-7ab5-42a4-97a4-b8f7cd34fe65 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.877215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.877841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:48.878162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.880181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.895396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 786/1564] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:09:48.897887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.899363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 779/1565] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:48.900765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.901273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.901763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.921998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b1b5a4-4bbf-4761-80c0-56714ef4267d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.930448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.931493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.932174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.932715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.933938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:48.934462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:48.937656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:48.937924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:48.939220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:48.949320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 780/1566] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:48.954395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.963672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 787/1567] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:09:48.965358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:48.974664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac27487a-677c-4dbf-891d-7dec629bd71f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.982653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.983637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.984094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.984542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] PATH_INFO: `/v3/projects/35820b4f79774ac49f43fce915f56d6f/users/e65f0f1ba54f483abff788974b1256cb/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.986355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9627f6f-c812-4d16-875b-03dbcc0905b3 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:48.995628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:48.996651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:48.997339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:48.997776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:48.999250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.003386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.009835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35820b4f79774ac49f43fce915f56d6f, user_id=e65f0f1ba54f483abff788974b1256cb, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.011962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:09:49.011962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:09:49.011962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:09:49.016135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9de63df-f682-4ae7-a2ea-7e1b20be6f33 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.026823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:49.030662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:49.043148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 781/1568] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:48 2022] PUT /identity/v3/projects/35820b4f79774ac49f43fce915f56d6f/users/e65f0f1ba54f483abff788974b1256cb/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:49.046081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.046668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.047160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.058814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 788/1569] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:48 2022] POST /identity/v3/projects => generated 377 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.061223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.088150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:49.090200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:49.174513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ee702-b734-44f6-89dc-5d95606fd8ae admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.180747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.182097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.182718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.183366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.184683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:49.185330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.188627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.193689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 782/1570] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:49.195081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.209283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082ef9d-18e2-4975-8835-05154e958755 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.221323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 789/1571] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:09:49.221611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.221611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.221611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.221611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.222163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.222293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.222293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca550b76-9cff-4d17-80b0-9012f0029c50 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.247955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.248896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.249337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.249752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] PATH_INFO: `/v3/projects/a0dde1f1f3c640c0aa8311bbf1de2fbe/users/d4925ac45b1b480c95da85ec4c7ab106/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.269868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 783/1572] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/projects => generated 377 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.271562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.278382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0dde1f1f3c640c0aa8311bbf1de2fbe, user_id=d4925ac45b1b480c95da85ec4c7ab106, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.284094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.305909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 790/1573] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:49 2022] PUT /identity/v3/projects/a0dde1f1f3c640c0aa8311bbf1de2fbe/users/d4925ac45b1b480c95da85ec4c7ab106/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:49.308070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.383526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7491c-92fb-4913-b9a2-3fa4c5c4b395 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.433292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dd59b3-7566-4e60-9385-4f21ddf8781f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.441443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510aa5a9-16f0-45c9-b91e-c39d98c74879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.442224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510aa5a9-16f0-45c9-b91e-c39d98c74879 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.442684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510aa5a9-16f0-45c9-b91e-c39d98c74879 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.443124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510aa5a9-16f0-45c9-b91e-c39d98c74879 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.444306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510aa5a9-16f0-45c9-b91e-c39d98c74879 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.447841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510aa5a9-16f0-45c9-b91e-c39d98c74879 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.479143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.479717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.480012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.480482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.481274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:49.481747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.483589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.495971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 791/1574] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/projects => generated 377 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.497700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc82644-07c4-477a-940a-d40b083d370a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.498011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc82644-07c4-477a-940a-d40b083d370a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.498296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc82644-07c4-477a-940a-d40b083d370a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.502872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 784/1575] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:09:49.505053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.530380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7d52eb-f5bd-4f6c-b75c-a9fd75ed3c96 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.538860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-afc82644-07c4-477a-940a-d40b083d370a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:49.540941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-afc82644-07c4-477a-940a-d40b083d370a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:49.556479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:49.626693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:49.639705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.640860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.641149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.641501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.642609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.646951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.656754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 792/1576] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:49.658312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc82644-07c4-477a-940a-d40b083d370a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.677660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc82644-07c4-477a-940a-d40b083d370a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.686718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 785/1577] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/users => generated 289 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.686718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.687911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.689039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.690453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.690453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.691407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:09:49.692000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.695156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.700494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14d02e2-706f-4073-93f3-a047b2cc6cb3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.706707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cd4f70-bbd9-43ff-ac18-5f74e84682e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.708293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cd4f70-bbd9-43ff-ac18-5f74e84682e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.708607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cd4f70-bbd9-43ff-ac18-5f74e84682e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.708883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cd4f70-bbd9-43ff-ac18-5f74e84682e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.709626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cd4f70-bbd9-43ff-ac18-5f74e84682e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.711715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27cd4f70-bbd9-43ff-ac18-5f74e84682e9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.718898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 793/1578] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:09:49.720495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.743776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512b7e1-c786-4c8a-9712-fc81d574f9d0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.752443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.753521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.753662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 786/1579] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/projects => generated 387 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.754019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.755029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.756196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.756634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.757067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.757653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.759603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.792952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 794/1580] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/users => generated 289 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.794781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.811319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b94fe4-0734-4174-8135-ebbe86cbd41e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.813191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] MFA Rules not processed for user `d4925ac45b1b480c95da85ec4c7ab106`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:49.814748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:49.918348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.919407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.919865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.920314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.921116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.923616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:49.949115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 787/1581] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:49.949382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 795/1582] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/users => generated 289 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:49.950269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.950456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:49.965021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d926803-c8ac-4730-97ae-a09762cf7a05 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:49.972770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:49.974788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:49.975263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:49.975719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:49.977072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:49.980867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.042325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 788/1583] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:49 2022] POST /identity/v3/projects => generated 383 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.042325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.066325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb0064-36fc-4bfe-8a18-d063eff302dc admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.139072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:50.144004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.146735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.147167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.147665 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.157633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e28dc39-aa46-4013-8bda-4b5750598baa admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.164475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.165421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.165835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.166009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.166291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.167095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.168678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.169505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.183321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 796/1584] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:50.184306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.193392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 789/1585] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.193392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.201047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2507a3e2-a407-4275-bed5-4e5b8268fee8 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.208151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.209118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.209627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.210078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.210892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.213661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.214750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a084dca-0c95-4961-88ed-81b234cc5c6d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.222813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:50.224529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.225507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.225989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.226522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.227914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.234448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.270341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 790/1586] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:50 2022] POST /identity/v3/users => generated 294 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.271684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.271906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 797/1587] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:50 2022] POST /identity/v3/projects => generated 367 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.273565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.295142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3390b0-86c9-4ddd-a21e-ed88e7b4496a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.395346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ff3938-6316-456a-8455-44f208c8d5d4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.406575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.407652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.408227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.408749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.410133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.414188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.415608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.416691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.417366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.418213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.419223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.421629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.437249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 791/1588] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.439073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.453514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4443ed-8d28-43cf-aebb-8b1db81b5a59 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.462651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.463900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.464458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.464934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.466183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.466500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 798/1589] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:50 2022] POST /identity/v3/projects => generated 365 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.467549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.469826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.492660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 792/1590] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.494557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.518006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1629830-c82c-446e-9187-e935e60d6b5e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.567321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c906465-9d90-4a2b-81b3-3d4a20a9cc94 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.575007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.575848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.576294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.576589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.577285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.579540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.607968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 799/1591] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:50 2022] POST /identity/v3/users => generated 292 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.608858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.620392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dd56dc-eb35-462b-be67-2dddd70da276 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.626118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.626745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.627697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.627697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.628042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.630341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.642768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 800/1592] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.643848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.654531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.655855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.656045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f5a201-8b68-4a5c-a043-b3a2ae7eba4f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.656382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.656866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.658641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.661525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.661898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.662258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.662515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.662799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.663583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.665763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.689467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 793/1593] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.689467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.697097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 801/1594] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:50 2022] POST /identity/v3/users => generated 284 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.697097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.709014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9566cb74-5d01-4de1-b1e4-74d7b8f84157 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.714846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd8d764-73f7-4664-b9c2-764220a87913 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.716959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.718293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.718612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.719242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.721232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.728325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.728452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.728551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.728551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.728551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.729481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.733066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.758113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 794/1595] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.765148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.767012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 802/1596] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:50 2022] POST /identity/v3/users => generated 289 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:50.768526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.783285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50785ca4-d27a-4fb4-bda4-a364a1908da1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.791709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245b4e07-59fe-4b4a-b52d-8bfef0d24b0b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.792208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.793713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.794347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.794861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.796579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.799618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.800876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.802918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.803589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.805037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.807568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.810633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.821914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 803/1597] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.823497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.830911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:50.848751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 795/1598] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.848751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.856223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e0b1d3-7d44-40a3-8d00-c21d1e6abe73 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.862688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.864864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.865457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.865988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.867103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.868655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39007df-dca4-46d6-b660-84e606f7bacf admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.869782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.877757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.878935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.879523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.880001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] PATH_INFO: `/v3/projects/7fd9ab6d98a44605a57bd8ae9535a686/users/4ae9c7ef17b746fc8f87e3c22a241b76/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.884924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 804/1599] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.886435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.904718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b298759b-c082-4eda-ab57-08df4a38ee1b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.914575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:50.915846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:50.916550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:50.917103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:50.917310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fd9ab6d98a44605a57bd8ae9535a686, user_id=4ae9c7ef17b746fc8f87e3c22a241b76, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.918645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:50.923213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.924434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:50.951894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 805/1600] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:50.953548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:50.972839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c287f5e4-1236-498f-a12a-61473362be4f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:50.976599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 796/1601] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:50 2022] PUT /identity/v3/projects/7fd9ab6d98a44605a57bd8ae9535a686/users/4ae9c7ef17b746fc8f87e3c22a241b76/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:50.978888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.008016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:51.104450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.105623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.106095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.106604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] PATH_INFO: `/v3/projects/a105c95deb1041ba9ac721d7c82110e9/users/caafb1ccfd704e3ca65b6c4db04b05ef/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.119203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d20ca4-2efd-4482-99f4-d11a3cd8219c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.126277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.127181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.127813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.129292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.130436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.132868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.141741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] RBAC: Authorizing `identity:create_grant(project_id=a105c95deb1041ba9ac721d7c82110e9, user_id=caafb1ccfd704e3ca65b6c4db04b05ef, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.147800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.151846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 797/1602] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:50 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:51.153545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.170590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a92849e-a605-49fc-8f3c-7813541ddf2a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.181189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 806/1603] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:50 2022] PUT /identity/v3/projects/a105c95deb1041ba9ac721d7c82110e9/users/caafb1ccfd704e3ca65b6c4db04b05ef/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:51.183018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.295385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.297082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.297082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.297691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.300365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.303210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.308400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6327a829-3696-4c3c-a6e2-d6dc1c31eaee admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.326121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.328165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.329211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.329922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] PATH_INFO: `/v3/projects/cd7b4755285e47588730fc262e7ccbdc/users/18725d3a879344bd9e79faed34f5724e/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.334483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 798/1604] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:51 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:51.336076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.355488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2196f9c6-f86c-4448-bf4d-a0718a4e60ec admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.364140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5911a27f-c945-47cb-ac22-55100c7c5bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.364632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd7b4755285e47588730fc262e7ccbdc, user_id=18725d3a879344bd9e79faed34f5724e, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.365495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911a27f-c945-47cb-ac22-55100c7c5bf2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.365877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911a27f-c945-47cb-ac22-55100c7c5bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.366409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5911a27f-c945-47cb-ac22-55100c7c5bf2 admin admin] PATH_INFO: `/v3/projects/1b1d33b8c71e41e6bf23d4ad8590d2c2/users/5fafe42957914b929f9ffee53ab6e7ac/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.370247 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.392126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5911a27f-c945-47cb-ac22-55100c7c5bf2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b1d33b8c71e41e6bf23d4ad8590d2c2, user_id=5fafe42957914b929f9ffee53ab6e7ac, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.398250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 807/1605] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:51 2022] PUT /identity/v3/projects/cd7b4755285e47588730fc262e7ccbdc/users/18725d3a879344bd9e79faed34f5724e/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:51.399427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5911a27f-c945-47cb-ac22-55100c7c5bf2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.399725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.432362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 799/1606] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:51 2022] PUT /identity/v3/projects/1b1d33b8c71e41e6bf23d4ad8590d2c2/users/5fafe42957914b929f9ffee53ab6e7ac/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:51.435175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.435710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.436305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.474195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:51.476517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:51.549457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:51.585967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d96d52d-1841-427c-955e-fb1040ed6ba1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.677635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 800/1607] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:51.680777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.705355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3626685d-44b9-4374-9c45-10fae29e3db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.706298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3626685d-44b9-4374-9c45-10fae29e3db5 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.706803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3626685d-44b9-4374-9c45-10fae29e3db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.707491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3626685d-44b9-4374-9c45-10fae29e3db5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.708599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3626685d-44b9-4374-9c45-10fae29e3db5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.711806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3626685d-44b9-4374-9c45-10fae29e3db5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.713130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a49e7-9920-48da-85c3-18670dfa8454 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.724969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610483d-8f49-4ca4-8299-48eed95bc9cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.727003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610483d-8f49-4ca4-8299-48eed95bc9cc admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.727527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610483d-8f49-4ca4-8299-48eed95bc9cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.728289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610483d-8f49-4ca4-8299-48eed95bc9cc admin admin] PATH_INFO: `/v3/projects/92c9ed1704b74c008fcb4c06e9430f71/users/652abba5f7fb460b9dfd5c46c12a8329/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.734925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 808/1608] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:51 2022] GET /identity/v3/roles => generated 1420 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:51.737465 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.737955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.739138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.758447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:51.771800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8610483d-8f49-4ca4-8299-48eed95bc9cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=92c9ed1704b74c008fcb4c06e9430f71, user_id=652abba5f7fb460b9dfd5c46c12a8329, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:51.774100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:51.776349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:51.782641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8610483d-8f49-4ca4-8299-48eed95bc9cc admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:51.805245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 809/1609] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:51.808063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:51.816890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 801/1610] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:51 2022] PUT /identity/v3/projects/92c9ed1704b74c008fcb4c06e9430f71/users/652abba5f7fb460b9dfd5c46c12a8329/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:51.821224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf012875-5d5b-4753-be34-46875561852b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.821922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf012875-5d5b-4753-be34-46875561852b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.822526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf012875-5d5b-4753-be34-46875561852b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:51.828909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf5b933-8a18-4fb3-9756-f40659a80447 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:51.901143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cf012875-5d5b-4753-be34-46875561852b None None] MFA Rules not processed for user `18725d3a879344bd9e79faed34f5724e`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:51.903484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cf012875-5d5b-4753-be34-46875561852b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:51.963064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba50543-8a87-4ce2-b9e0-741102e30bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:51.965523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba50543-8a87-4ce2-b9e0-741102e30bda admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:51.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba50543-8a87-4ce2-b9e0-741102e30bda admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:51.966931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba50543-8a87-4ce2-b9e0-741102e30bda admin admin] PATH_INFO: `/v3/projects/fce127d62e6648d89d54ab21bd3b14e2/users/c59b392a2e1a4f32bba1433afc519d5e/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.006762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba50543-8a87-4ce2-b9e0-741102e30bda admin admin] RBAC: Authorizing `identity:create_grant(project_id=fce127d62e6648d89d54ab21bd3b14e2, user_id=c59b392a2e1a4f32bba1433afc519d5e, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:52.017584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba50543-8a87-4ce2-b9e0-741102e30bda admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:52.054254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 810/1611] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:51 2022] PUT /identity/v3/projects/fce127d62e6648d89d54ab21bd3b14e2/users/c59b392a2e1a4f32bba1433afc519d5e/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:52.058134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.059004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.059973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.063122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 802/1612] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:52.064458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf012875-5d5b-4753-be34-46875561852b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:52.107398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:52.110927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:52.212061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf012875-5d5b-4753-be34-46875561852b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:52.288025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 811/1613] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:52.294727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:52.342302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033668cc-82da-4cc5-9f23-6033a2de52e3 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:52.351747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:52.352867 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.353317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.353848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] PATH_INFO: `/v3/projects/cb5a2700217d4324bedb0014e43804db/users/2a41d2a303b44f41b22cc6139aded533/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.359599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:52.375666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:52.376864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.377374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.377887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.391570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb5a2700217d4324bedb0014e43804db, user_id=2a41d2a303b44f41b22cc6139aded533, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:52.398709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:52.399602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:52.403238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:52.431063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 812/1614] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:52 2022] PUT /identity/v3/projects/cb5a2700217d4324bedb0014e43804db/users/2a41d2a303b44f41b22cc6139aded533/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:52.432445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 803/1615] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:52.432852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:52.436473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:52.455230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5baa821-68e9-4a47-96aa-4bde1383c219 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:52.541835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:52.587765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b1bedf-77fb-4a1b-b8c8-33c0557d19b0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:52.596786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa0a42-33d4-4b90-bc70-9deee0d6c908 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:52.597853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa0a42-33d4-4b90-bc70-9deee0d6c908 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.598436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa0a42-33d4-4b90-bc70-9deee0d6c908 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.599014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa0a42-33d4-4b90-bc70-9deee0d6c908 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.601139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:52.601815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.602079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.602563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.604433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:52.606441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:52.624711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa0a42-33d4-4b90-bc70-9deee0d6c908 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:52.628659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa0a42-33d4-4b90-bc70-9deee0d6c908 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:52.634719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 804/1616] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:52 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:52.637324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:52.657509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 813/1617] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:52.666232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619b6b8f-93e8-4187-a971-e81e69491a9e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.666712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619b6b8f-93e8-4187-a971-e81e69491a9e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.667225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619b6b8f-93e8-4187-a971-e81e69491a9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.670790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cab606-a1b5-41fb-bca1-4db28669c73d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:52.684115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:52.685224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:52.685688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:52.686144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:52.687257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:52.691554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:52.757453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-619b6b8f-93e8-4187-a971-e81e69491a9e None None] MFA Rules not processed for user `c59b392a2e1a4f32bba1433afc519d5e`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:52.760356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-619b6b8f-93e8-4187-a971-e81e69491a9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:52.785594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 805/1618] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:09:52 2022] POST /identity/v3/projects => generated 377 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:52.787260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:53.000620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a90ebc-e2c2-464d-b176-a49c45b5fe13 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:53.014222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 814/1619] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 352 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:53.014222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.014222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.017096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:53.018331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.019878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.020199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.026920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.166090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] MFA Rules not processed for user `2a41d2a303b44f41b22cc6139aded533`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:53.170407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:53.253167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:53.272504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:53.319116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 806/1620] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:53.320901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:53.347579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746f1ae2-ad4c-45c6-921e-ef032db40a38 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:53.363864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5504fd4-7c60-4966-9aa6-0600704b9ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:53.365120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5504fd4-7c60-4966-9aa6-0600704b9ea4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.366518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5504fd4-7c60-4966-9aa6-0600704b9ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.367254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5504fd4-7c60-4966-9aa6-0600704b9ea4 admin admin] PATH_INFO: `/v3/projects/92c9ed1704b74c008fcb4c06e9430f71/users/652abba5f7fb460b9dfd5c46c12a8329/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.376088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5504fd4-7c60-4966-9aa6-0600704b9ea4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92c9ed1704b74c008fcb4c06e9430f71, user_id=652abba5f7fb460b9dfd5c46c12a8329, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:53.379985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5504fd4-7c60-4966-9aa6-0600704b9ea4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:53.399067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 815/1621] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:53.402036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:53.418920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:53.428221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64572873-f8b3-4331-a91a-5e3e2d713f9c None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:53.440844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 807/1622] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:53 2022] PUT /identity/v3/projects/92c9ed1704b74c008fcb4c06e9430f71/users/652abba5f7fb460b9dfd5c46c12a8329/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:53.445062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444596b8-436c-4d15-8f3b-d2fd59792a65 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.446025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444596b8-436c-4d15-8f3b-d2fd59792a65 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.446754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444596b8-436c-4d15-8f3b-d2fd59792a65 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.450712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 808/1623] 10.222.0.26 () {54 vars in 962 bytes} [Sun Aug 7 01:09:53 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:09:53.452926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444596b8-436c-4d15-8f3b-d2fd59792a65 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:53.579054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9416d9be-09dd-4adb-9c05-35cf6d3bc733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:53.582016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416d9be-09dd-4adb-9c05-35cf6d3bc733 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.582816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416d9be-09dd-4adb-9c05-35cf6d3bc733 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.583518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9416d9be-09dd-4adb-9c05-35cf6d3bc733 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.593422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9416d9be-09dd-4adb-9c05-35cf6d3bc733 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:53.599168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9416d9be-09dd-4adb-9c05-35cf6d3bc733 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:53.615594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444596b8-436c-4d15-8f3b-d2fd59792a65 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:53.623232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7869a232-a86d-41f9-b729-50bc68878545 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:53.624756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7869a232-a86d-41f9-b729-50bc68878545 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.625255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7869a232-a86d-41f9-b729-50bc68878545 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.625798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7869a232-a86d-41f9-b729-50bc68878545 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.635259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:53.672767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 816/1624] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:09:53 2022] POST /identity/v3/users => generated 295 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:09:53.672767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.672767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.673413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.729403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7869a232-a86d-41f9-b729-50bc68878545 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:53.731335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7869a232-a86d-41f9-b729-50bc68878545 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:53.746865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 809/1625] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:53.748992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:53.749399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:53.749849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:53.841530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] MFA Rules not processed for user `652abba5f7fb460b9dfd5c46c12a8329`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:53.850373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:53.953200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:53.954614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:54.239703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 810/1626] 10.222.0.26 () {58 vars in 1059 bytes} [Sun Aug 7 01:09:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 492 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:54.241346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:54.294380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 817/1627] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:53 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 625 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:54.315645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:54.328311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e79d14-7ab9-4234-bf15-51cbcf36ac35 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:54.361095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:54.362191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:54.362651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:54.363133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:54.365009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:54.372174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:54.388100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf335983-51ce-4b75-a422-c25392b97fb0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:54.420375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:54.426175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 811/1628] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:54 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:54.427846 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:54.428021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:54.430769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:54.431398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:54.474959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cae648-ba9e-43d5-a6ae-ad28e3e60fb2 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:54.488432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:54.489536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:54.490048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:54.490512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:54.524279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:54.528601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:54.594870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 812/1629] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:54 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:54.603785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:54.654372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027a84e-e1aa-4e02-97bf-c2000d9d7050 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:54.682398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:54.684202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:54.686237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:54.687087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] PATH_INFO: `/v3/projects/07d5d8a81c374d10ba8ef88e23aa31f5/users/40cfa432fe8141f18c8a83a162f08a2c/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:54.728691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07d5d8a81c374d10ba8ef88e23aa31f5, user_id=40cfa432fe8141f18c8a83a162f08a2c, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:54.735649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:54.776423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 813/1630] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:54 2022] PUT /identity/v3/projects/07d5d8a81c374d10ba8ef88e23aa31f5/users/40cfa432fe8141f18c8a83a162f08a2c/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:54.782183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:54.829436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:54.841021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:54.893795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 818/1631] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:54 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 598 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:54.937795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:55.049075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2b490-a07d-4b00-8df4-27d381c6b7f5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:55.051281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:55.067851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:55.069177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:55.071713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:55.073457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:55.078621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f62aa-1a37-442d-aca2-496339b1be67 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:55.087006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:55.099289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:55.149131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:55.174140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 814/1632] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:09:54 2022] GET /identity/v3/roles => generated 1420 bytes in 388 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:09:55.189830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:55.255525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae2d712-9290-4ad1-83b3-74768b9a7bc3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:55.280815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78b9ff-9ff4-4a20-a674-1abdc5bf240f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:55.280815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78b9ff-9ff4-4a20-a674-1abdc5bf240f admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:55.280815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78b9ff-9ff4-4a20-a674-1abdc5bf240f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:55.280815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78b9ff-9ff4-4a20-a674-1abdc5bf240f admin admin] PATH_INFO: `/v3/projects/07d5d8a81c374d10ba8ef88e23aa31f5/users/40cfa432fe8141f18c8a83a162f08a2c/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:55.289348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa78b9ff-9ff4-4a20-a674-1abdc5bf240f admin admin] RBAC: Authorizing `identity:create_grant(project_id=07d5d8a81c374d10ba8ef88e23aa31f5, user_id=40cfa432fe8141f18c8a83a162f08a2c, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:55.305397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa78b9ff-9ff4-4a20-a674-1abdc5bf240f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:55.364271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 815/1633] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:09:55 2022] PUT /identity/v3/projects/07d5d8a81c374d10ba8ef88e23aa31f5/users/40cfa432fe8141f18c8a83a162f08a2c/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:09:55.372634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:55.373137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:55.373639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:55.398227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4b55ef-546d-4c05-bc41-0b4359dfd541 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:55.398891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4b55ef-546d-4c05-bc41-0b4359dfd541 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:55.399177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4b55ef-546d-4c05-bc41-0b4359dfd541 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:55.399460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4b55ef-546d-4c05-bc41-0b4359dfd541 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:55.523989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] MFA Rules not processed for user `40cfa432fe8141f18c8a83a162f08a2c`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:55.531450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:55.660600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4b55ef-546d-4c05-bc41-0b4359dfd541 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:55.663445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4b55ef-546d-4c05-bc41-0b4359dfd541 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:55.707752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 819/1634] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:55 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 656 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:55.939913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fc8420-d67a-4e6c-91e2-e1fc8eb92d1f None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:55.942601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fc8420-d67a-4e6c-91e2-e1fc8eb92d1f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:55.943250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fc8420-d67a-4e6c-91e2-e1fc8eb92d1f None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:55.947328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 820/1635] 10.222.0.26 () {54 vars in 912 bytes} [Sun Aug 7 01:09:55 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:09:55.963910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 816/1636] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:09:55 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 592 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:55.965188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:55.965799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:55.970314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:56.066480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:56.069952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:56.218143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:56.283056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 821/1637] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:09:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:56.411513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:56.453114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97356f36-feb5-4d80-b811-d6bb5275df05 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:56.472613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f108c91e-9dfb-4933-9825-a7b8b01bc708 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:56.473834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f108c91e-9dfb-4933-9825-a7b8b01bc708 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:56.474320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f108c91e-9dfb-4933-9825-a7b8b01bc708 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:56.474806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f108c91e-9dfb-4933-9825-a7b8b01bc708 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:56.544079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def58df-b8a5-4995-9744-2d9f5a5837c0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:56.549129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45199bec-c74a-404b-b0ee-ad7f9589e09e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:56.550420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45199bec-c74a-404b-b0ee-ad7f9589e09e service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:56.550704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45199bec-c74a-404b-b0ee-ad7f9589e09e service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:56.551145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45199bec-c74a-404b-b0ee-ad7f9589e09e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:56.565703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45199bec-c74a-404b-b0ee-ad7f9589e09e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:56.570812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45199bec-c74a-404b-b0ee-ad7f9589e09e service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:56.590306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 822/1638] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:09:56.679971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f108c91e-9dfb-4933-9825-a7b8b01bc708 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:56.679971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f108c91e-9dfb-4933-9825-a7b8b01bc708 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:56.688262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:56.720627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 817/1639] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:09:58.575973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d776739-ecb1-416d-971d-5e3087bf1a10 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:58.577266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d776739-ecb1-416d-971d-5e3087bf1a10 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:58.577389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d776739-ecb1-416d-971d-5e3087bf1a10 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:58.581643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 823/1640] 10.222.0.26 () {54 vars in 962 bytes} [Sun Aug 7 01:09:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:09:58.593587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a72e96-7a79-45ca-a087-dbb988d51dac None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:58.594388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a72e96-7a79-45ca-a087-dbb988d51dac None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:58.594487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a72e96-7a79-45ca-a087-dbb988d51dac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:58.649226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-59a72e96-7a79-45ca-a087-dbb988d51dac None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:58.652470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-59a72e96-7a79-45ca-a087-dbb988d51dac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:58.676925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 818/1641] 10.222.0.26 () {58 vars in 1059 bytes} [Sun Aug 7 01:09:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:58.679245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80fddd5-8bba-48f6-9a95-249bde77b9a0 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:58.679726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80fddd5-8bba-48f6-9a95-249bde77b9a0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:58.680167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80fddd5-8bba-48f6-9a95-249bde77b9a0 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:58.683716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 824/1642] 10.222.0.26 () {54 vars in 912 bytes} [Sun Aug 7 01:09:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:09:58.696664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:58.697234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:58.697887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:58.729899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:09:58.732389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:09:58.747585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 819/1643] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:09:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:09:58.777535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80fddd5-8bba-48f6-9a95-249bde77b9a0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:09:58.829090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80fddd5-8bba-48f6-9a95-249bde77b9a0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:09:58.845476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:09:58.846396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:09:58.846870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:09:58.847521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:09:58.875574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:09:58.893331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:09:58.896171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:09:58.960366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 825/1644] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:09:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:03.686238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:03.717930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f486c4db-6ea3-43e6-8bf9-d9bbdab7b3e1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:03.744362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6ba278-81f4-4a08-8380-712b16161559 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:03.744362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6ba278-81f4-4a08-8380-712b16161559 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:03.744362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6ba278-81f4-4a08-8380-712b16161559 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:03.744362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6ba278-81f4-4a08-8380-712b16161559 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:03.839016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6ba278-81f4-4a08-8380-712b16161559 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:03.844410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6ba278-81f4-4a08-8380-712b16161559 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:03.868445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 820/1645] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:03 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:04.857000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:05.088872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e7e16-9c67-4b35-aa81-e5b686e638b2 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:05.104726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:05.107156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:05.109670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:05.110389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] PATH_INFO: `/v3/users/d4925ac45b1b480c95da85ec4c7ab106` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:05.114354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4925ac45b1b480c95da85ec4c7ab106)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:05.121727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:05.309100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4925ac45b1b480c95da85ec4c7ab106'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:05.362577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4925ac45b1b480c95da85ec4c7ab106'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:05.403398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4925ac45b1b480c95da85ec4c7ab106'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:05.406678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4925ac45b1b480c95da85ec4c7ab106'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:05.409188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4925ac45b1b480c95da85ec4c7ab106'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:05.460802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 826/1646] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:04 2022] DELETE /identity/v3/users/d4925ac45b1b480c95da85ec4c7ab106 => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:05.467121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b108663-ba30-452c-be25-1935d8a404f8 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:05.469941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b108663-ba30-452c-be25-1935d8a404f8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:05.470437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b108663-ba30-452c-be25-1935d8a404f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:05.512485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1b108663-ba30-452c-be25-1935d8a404f8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:05.514511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1b108663-ba30-452c-be25-1935d8a404f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:05.774484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 821/1647] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:05.828044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:05.830016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:06.019470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cc9d1f-0437-4d18-9cd3-37a35da16623 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:06.028635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:06.029445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:06.031448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:06.031448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:06.050214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:06.055222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:06.074514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 827/1648] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:06.602616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:06.603328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:06.604182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:06.818050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] MFA Rules not processed for user `5fafe42957914b929f9ffee53ab6e7ac`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:06.825504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:07.091858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:07.094555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:07.116156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 822/1649] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:06 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 516 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:07.142941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c12775-64cf-404f-988f-f45d87cc69b8 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:07.156803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:07.159203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:07.176829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:07.178345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:07.179723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:07.180846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] PATH_INFO: `/v3/projects/a0dde1f1f3c640c0aa8311bbf1de2fbe` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:07.189634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0dde1f1f3c640c0aa8311bbf1de2fbe)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:07.198670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: WARNING py.warnings [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:10:07.198670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: warnings.warn(msg) Aug 07 01:10:07.198670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: Aug 07 01:10:07.208040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:07.297438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:07.423502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:07.461992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff659bcb-ed41-4171-9bd6-cc7703d6e117 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:07.496813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:07.497370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:07.497659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:07.497967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:07.529614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:07.531456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:07.580614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0dde1f1f3c640c0aa8311bbf1de2fbe', 'request_id': 'req-01f9f645-f8a4-4165-a404-ae60fafb8d91'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:07.610183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0dde1f1f3c640c0aa8311bbf1de2fbe', 'request_id': 'req-01f9f645-f8a4-4165-a404-ae60fafb8d91'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:07.617484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 823/1650] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:07 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:10:07.639288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0dde1f1f3c640c0aa8311bbf1de2fbe', 'request_id': 'req-01f9f645-f8a4-4165-a404-ae60fafb8d91'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:07.731287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] The token cache is being invalidate because project a0dde1f1f3c640c0aa8311bbf1de2fbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:07.731903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:07.734481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:07.740158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 828/1651] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:07 2022] DELETE /identity/v3/projects/a0dde1f1f3c640c0aa8311bbf1de2fbe => generated 0 bytes in 650 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:10:07.746926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:07.783193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752ae365-e623-4422-837c-c201f0eff931 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:08.086030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:08.088477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:08.088955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:08.089429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] PATH_INFO: `/v3/users/652abba5f7fb460b9dfd5c46c12a8329` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:08.094831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] RBAC: Authorizing `identity:delete_user(user_id=652abba5f7fb460b9dfd5c46c12a8329)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:08.109172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: WARNING py.warnings [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 07 01:10:08.109172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: warnings.warn(msg) Aug 07 01:10:08.109172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: Aug 07 01:10:08.111632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:08.333840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '652abba5f7fb460b9dfd5c46c12a8329'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:08.335028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '652abba5f7fb460b9dfd5c46c12a8329'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:08.336078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '652abba5f7fb460b9dfd5c46c12a8329'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:08.359219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '652abba5f7fb460b9dfd5c46c12a8329'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:08.386874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-896ca819-bcf9-43e6-968a-2be2ecaa9629 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '652abba5f7fb460b9dfd5c46c12a8329'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:08.494483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 824/1652] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:07 2022] DELETE /identity/v3/users/652abba5f7fb460b9dfd5c46c12a8329 => generated 0 bytes in 749 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:10.730719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:10.732541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:10.791425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9f645-f8a4-4165-a404-ae60fafb8d91 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:11.076671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:11.081770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:11.082835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:11.083644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] PATH_INFO: `/v3/projects/92c9ed1704b74c008fcb4c06e9430f71` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:11.091700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92c9ed1704b74c008fcb4c06e9430f71)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:11.126742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:11.482466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92c9ed1704b74c008fcb4c06e9430f71', 'request_id': 'req-862ba997-0444-4a95-82e2-8f71cf5576d0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:11.531464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92c9ed1704b74c008fcb4c06e9430f71', 'request_id': 'req-862ba997-0444-4a95-82e2-8f71cf5576d0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:11.544548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92c9ed1704b74c008fcb4c06e9430f71', 'request_id': 'req-862ba997-0444-4a95-82e2-8f71cf5576d0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:11.586599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] The token cache is being invalidate because project 92c9ed1704b74c008fcb4c06e9430f71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:11.586952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:11.589990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-862ba997-0444-4a95-82e2-8f71cf5576d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:11.602902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 829/1653] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:10 2022] DELETE /identity/v3/projects/92c9ed1704b74c008fcb4c06e9430f71 => generated 0 bytes in 867 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:11.604147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:11.605575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:11.606259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:11.702313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:11.704307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:11.900289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 825/1654] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:11.906594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcbbf52-7808-4311-9adf-b4bf8a275ca8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:11.908300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcbbf52-7808-4311-9adf-b4bf8a275ca8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:11.909310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcbbf52-7808-4311-9adf-b4bf8a275ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.003848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2fcbbf52-7808-4311-9adf-b4bf8a275ca8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:12.006865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2fcbbf52-7808-4311-9adf-b4bf8a275ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:12.052469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 830/1655] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:12.058139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.187692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.188750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.190099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.198377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31661a8c-ac60-4a7c-b2f2-eb3c748ff45c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:12.216514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:12.216709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.218084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.218551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.219513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:12.220174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:12.223519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:12.229008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:12.286039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 826/1656] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:12.293590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.361291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da22cbaa-1c93-444d-bf73-6705abb4485a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:12.383496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] MFA Rules not processed for user `e65f0f1ba54f483abff788974b1256cb`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:12.385212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:12.396778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:12.402142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.403915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.405428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.417034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:12.427712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:12.429197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:12.535293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 827/1657] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:12 2022] POST /identity/v3/projects => generated 387 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:12.538156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.666413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf274c3-735b-4edc-ac99-4748897f4db1 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:12.682957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:12.683393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 831/1658] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 507 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:12.684099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.684545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.693683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.694753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:12.698132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:12.747381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.749268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:12.767062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 828/1659] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:12 2022] POST /identity/v3/users => generated 294 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:12.768573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.794406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa15c55-d87c-435b-b020-53e74bf34da4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:12.806445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:12.807091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.807374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.807646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.810865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:12.815310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:12.847876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 829/1660] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:12 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:12.848813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.866524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee0891d-c366-495d-bb64-641bd5595101 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:12.882278 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:12.883028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.886025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.886491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.887203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:12.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:12.922953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 830/1661] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:12 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:12.930480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:12.946258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e8d11b-5251-47e0-abdd-7fb4a3076370 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:12.961470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f715fa-7eaa-4612-a559-c1acfd1ecde9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:12.962534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f715fa-7eaa-4612-a559-c1acfd1ecde9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:12.962818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f715fa-7eaa-4612-a559-c1acfd1ecde9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:12.963113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f715fa-7eaa-4612-a559-c1acfd1ecde9 admin admin] PATH_INFO: `/v3/projects/4c5e5b3c09904d95bb215126b79de50f/users/398cd5e6f3a04ebfa62a5c61199c06e3/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:12.999584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f715fa-7eaa-4612-a559-c1acfd1ecde9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c5e5b3c09904d95bb215126b79de50f, user_id=398cd5e6f3a04ebfa62a5c61199c06e3, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:13.012176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f715fa-7eaa-4612-a559-c1acfd1ecde9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:13.055774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 831/1662] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:12 2022] PUT /identity/v3/projects/4c5e5b3c09904d95bb215126b79de50f/users/398cd5e6f3a04ebfa62a5c61199c06e3/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:13.067536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:13.067833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:13.068090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:13.106787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df55190-b60d-45ae-b389-04ddec49ce81 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:13.141252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] MFA Rules not processed for user `398cd5e6f3a04ebfa62a5c61199c06e3`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:13.148413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:13.400097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 832/1663] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:13 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:13.400097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:13.409890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:13.423759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:13.505594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:13.507902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:13.508500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:13.508996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:13.633861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863ad064-b8a3-4496-8adf-92be80c2f2dc None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:13.686388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:13.692402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:13.695416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:13.696293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] PATH_INFO: `/v3/users/c59b392a2e1a4f32bba1433afc519d5e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:13.698902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c59b392a2e1a4f32bba1433afc519d5e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:13.702053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:13.778950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:13.829099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:13.831910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:13.861735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 832/1664] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:12 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 1114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:13.866293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:13.902191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bfab9e-6ba5-4dbd-8504-9337e1aa2819 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:13.905738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c59b392a2e1a4f32bba1433afc519d5e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:13.907029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c59b392a2e1a4f32bba1433afc519d5e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:13.908029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c59b392a2e1a4f32bba1433afc519d5e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:13.918810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:13.919628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:13.919987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:13.921322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:13.924159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:13.926391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c59b392a2e1a4f32bba1433afc519d5e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:13.931268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:13.951679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-67b666bb-0952-433f-aed7-851825252c01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c59b392a2e1a4f32bba1433afc519d5e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:14.060036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 833/1665] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:13 2022] DELETE /identity/v3/users/c59b392a2e1a4f32bba1433afc519d5e => generated 0 bytes in 663 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:14.067532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:14.079089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 833/1666] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:13 2022] POST /identity/v3/projects => generated 385 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:14.088791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:14.091890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:14.093383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:14.097686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:14.183003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:14.196967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:14.357201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 834/1667] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:14.358254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:14.359855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:14.397838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0620649-6e20-4e3c-9c24-7422577aec20 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:14.402344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:14.412877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:14.414515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:14.415274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:14.416078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:14.420438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:14.425431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:14.459640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078ee4fa-9324-4014-8065-82f2f987ccce None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:14.464603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:14.465286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:14.465574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:14.465854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:14.478470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:14.480298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:14.495132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 835/1668] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:14.495987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 834/1669] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:14 2022] POST /identity/v3/users => generated 299 bytes in 413 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:10:14.507551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:14.543299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbab32ef-2d90-413a-b590-744e771a9d93 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:14.593281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:14.595272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:14.595589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:14.596326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:14.597555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:14.609406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:14.718752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 836/1670] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:14.726031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:14.768282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5c6372-5eb7-4c8a-b997-906a964ccece admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:14.790912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:14.794823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:14.796577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:14.797086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] PATH_INFO: `/v3/projects/d41642e6db5d49119944c297c84ddc38/users/05487dffe07d460a9620db919787350c/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:14.897690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d41642e6db5d49119944c297c84ddc38, user_id=05487dffe07d460a9620db919787350c, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:14.911033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:14.968988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 835/1671] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:14 2022] PUT /identity/v3/projects/d41642e6db5d49119944c297c84ddc38/users/05487dffe07d460a9620db919787350c/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:14.971065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:15.055079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b86ec46-3da0-4e30-bddf-0f8ae689e4da admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:15.373223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:15.374300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:15.374795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:15.375279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:15.376395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:15.379224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:15.450805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:15.451576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 837/1672] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 481 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:15.496121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c81baf-3d64-4885-bf29-bc9b4d1edfb3 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:15.532146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0541c49-7860-41bf-ad16-d0c62262c4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:15.536358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0541c49-7860-41bf-ad16-d0c62262c4f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:15.536811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0541c49-7860-41bf-ad16-d0c62262c4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:15.547490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0541c49-7860-41bf-ad16-d0c62262c4f1 admin admin] PATH_INFO: `/v3/projects/d41642e6db5d49119944c297c84ddc38/users/05487dffe07d460a9620db919787350c/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:15.560341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0541c49-7860-41bf-ad16-d0c62262c4f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d41642e6db5d49119944c297c84ddc38, user_id=05487dffe07d460a9620db919787350c, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:15.566180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0541c49-7860-41bf-ad16-d0c62262c4f1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:15.610631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 836/1673] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:15 2022] PUT /identity/v3/projects/d41642e6db5d49119944c297c84ddc38/users/05487dffe07d460a9620db919787350c/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:15.623135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:15.626750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:15.634523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:15.635126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:15.635923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:15.688158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9bbf5b-1fd9-4ab4-9aad-aec429ceff08 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:15.704676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] MFA Rules not processed for user `05487dffe07d460a9620db919787350c`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:15.708008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:15.861911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:15.863004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:15.867004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:15.867512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] PATH_INFO: `/v3/projects/fce127d62e6648d89d54ab21bd3b14e2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:15.871574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fce127d62e6648d89d54ab21bd3b14e2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:15.882255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:15.983081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 837/1674] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:15 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:16.006775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:16.011545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:16.027979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:16.113419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fce127d62e6648d89d54ab21bd3b14e2', 'request_id': 'req-f94ba731-2b12-4325-89e3-3b9829a09d3f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.129736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:16.146829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fce127d62e6648d89d54ab21bd3b14e2', 'request_id': 'req-f94ba731-2b12-4325-89e3-3b9829a09d3f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.169982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fce127d62e6648d89d54ab21bd3b14e2', 'request_id': 'req-f94ba731-2b12-4325-89e3-3b9829a09d3f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.218118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] The token cache is being invalidate because project fce127d62e6648d89d54ab21bd3b14e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:16.219049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.222289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f94ba731-2b12-4325-89e3-3b9829a09d3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.231254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 838/1675] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:15 2022] DELETE /identity/v3/projects/fce127d62e6648d89d54ab21bd3b14e2 => generated 0 bytes in 609 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:16.239947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4d5230-771a-45db-b492-2c8337df97c7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:16.240463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4d5230-771a-45db-b492-2c8337df97c7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:16.241577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4d5230-771a-45db-b492-2c8337df97c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:16.261528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4186d5-2f1c-4b9b-b0ce-088d73a1ff78 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:16.322221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-aa4d5230-771a-45db-b492-2c8337df97c7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:16.326200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-aa4d5230-771a-45db-b492-2c8337df97c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:16.401606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:16.402605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:16.403044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:16.403584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:16.408456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:16.513479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 839/1676] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:16.528681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:16.529750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:16.530298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:16.583919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:16.594294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:16.601774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:16.606671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:16.637386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 838/1677] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:16 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 632 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:10:16.644293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 840/1678] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:16.647916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:16.681789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ab06b4-4a14-4999-9399-bc16d8afad8c service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:16.709368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:16.709901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:16.710894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:16.712002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:16.712185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:16.713243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:16.714854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:16.715526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:16.718853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:16.742515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0a5b86-30f3-4907-8e0c-c378ebe811db None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:16.762080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:16.762730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:16.763182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:16.763468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] PATH_INFO: `/v3/users/398cd5e6f3a04ebfa62a5c61199c06e3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:16.766839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 839/1679] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:16.767392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=398cd5e6f3a04ebfa62a5c61199c06e3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:16.769433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:16.774866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:16.804321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf4660e-3921-449f-b5b5-9cb6b3af9e80 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:16.818922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:16.826328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:16.826571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:16.828809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:16.833164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:16.838049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:16.902260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '398cd5e6f3a04ebfa62a5c61199c06e3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.904645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '398cd5e6f3a04ebfa62a5c61199c06e3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.913370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '398cd5e6f3a04ebfa62a5c61199c06e3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.939715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 840/1680] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:16 2022] POST /identity/v3/projects => generated 405 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:16.942632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:16.944920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '398cd5e6f3a04ebfa62a5c61199c06e3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:16.948670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:16.997967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-97f71407-0d2b-4280-bfe8-f5c910089e79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '398cd5e6f3a04ebfa62a5c61199c06e3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:17.122536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 841/1681] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:16 2022] DELETE /identity/v3/users/398cd5e6f3a04ebfa62a5c61199c06e3 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:17.133530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:17.134726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:17.139401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:17.185953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:17.205609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:17.209339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:17.264375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdb1c05-99e4-4130-97f6-52df381e7933 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:17.346157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:17.448878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 842/1682] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:17.459261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:17.462161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:17.529468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:17.532488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:17.532826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:17.533035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:17.533802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:17.542929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:17.627327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 841/1683] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:16 2022] POST /identity/v3/users => generated 303 bytes in 687 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:17.633114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:17.654850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e9ff1f-5aec-4b03-96af-f67db2be7d16 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:17.663376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:17.664643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:17.665368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:17.666254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:17.689223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883f2257-133d-40fe-b9af-d6c182fc0e28 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:17.696335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:17.699374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:17.712840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:17.714060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:17.714541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:17.715320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:17.716019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:17.745266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 843/1684] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:17.753364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:17.814025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 842/1685] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:17 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:10:17.823061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:17.843542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f343ca-7a37-4b37-8a63-1c80051d896a service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:17.850982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5136caa3-c233-4547-8e2f-f9c302210159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:17.851797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136caa3-c233-4547-8e2f-f9c302210159 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:17.851967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136caa3-c233-4547-8e2f-f9c302210159 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:17.852410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5136caa3-c233-4547-8e2f-f9c302210159 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:17.853095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5136caa3-c233-4547-8e2f-f9c302210159 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:17.855227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5136caa3-c233-4547-8e2f-f9c302210159 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:17.874363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 844/1686] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:17 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:17.878623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:17.914336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d87dbbc-7dd7-4d9b-a7ad-d6aac2364881 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:17.924357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:17.926482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:17.926806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:17.927319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] PATH_INFO: `/v3/projects/8aa8a1e6519c47f7b544945035640933/users/5f705757cfb2407e9e7d6487917bda7e/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:17.967233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8aa8a1e6519c47f7b544945035640933, user_id=5f705757cfb2407e9e7d6487917bda7e, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:17.971747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:18.007318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 843/1687] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:17 2022] PUT /identity/v3/projects/8aa8a1e6519c47f7b544945035640933/users/5f705757cfb2407e9e7d6487917bda7e/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:18.033394 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:18.034122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:18.034286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:18.138523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] MFA Rules not processed for user `5f705757cfb2407e9e7d6487917bda7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:18.141054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:18.362563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 845/1688] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:18.366887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:18.539299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff13203-3ada-4b15-9578-92901eb5f884 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:18.586571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2713f9-ef8d-4c8f-886e-bc37bf7b65f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:18.587227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2713f9-ef8d-4c8f-886e-bc37bf7b65f6 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:18.587595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2713f9-ef8d-4c8f-886e-bc37bf7b65f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:18.587874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2713f9-ef8d-4c8f-886e-bc37bf7b65f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:18.588576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2713f9-ef8d-4c8f-886e-bc37bf7b65f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:18.590748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2713f9-ef8d-4c8f-886e-bc37bf7b65f6 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:18.719915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 844/1689] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:18 2022] POST /identity/v3/projects => generated 405 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:10:18.724790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:18.759814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb7a2ee-ced0-4f33-ba69-5c77f8c1e5e3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:18.907895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:18.908494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:18.909074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:18.981909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] MFA Rules not processed for user `5fafe42957914b929f9ffee53ab6e7ac`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:18.985830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:18.985830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:18.986207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:18.986597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:18.987784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:18.989955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:18.990779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:19.026056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:19.042307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:19.062952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 846/1690] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:18 2022] POST /identity/v3/users => generated 309 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:19.069733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:19.120480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71670bc3-bbbc-4a5c-8fe5-50d2cc2ff77e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:19.148307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:19.154192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:19.154701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:19.159951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:19.164977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:19.177715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:19.236835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 847/1691] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:19.241916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:19.289146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 845/1692] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:18 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 382 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:19.304228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7957cbd-73aa-46b1-a20a-41fcf958c206 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:19.336562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:19.339493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:19.340048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:19.348944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] PATH_INFO: `/v3/projects/01199f8dbf6a441b9f2b6233a5ef1855/users/51bbd2b14c6b46d0ae846f973c088df2/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:19.439735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] RBAC: Authorizing `identity:create_grant(project_id=01199f8dbf6a441b9f2b6233a5ef1855, user_id=51bbd2b14c6b46d0ae846f973c088df2, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:19.471087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:19.559228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 848/1693] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:19 2022] PUT /identity/v3/projects/01199f8dbf6a441b9f2b6233a5ef1855/users/51bbd2b14c6b46d0ae846f973c088df2/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:19.564042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:19.624716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a47c006-4668-42b5-ba70-97142bad56fb None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:19.676780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:19.679817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:19.958980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:19.960289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:19.961035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:19.961708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:19.968405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:19.971605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:20.018399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54d8428-38d6-4d6e-9c49-cbf7caa0b64c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:20.032696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:20.033993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:20.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:20.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] PATH_INFO: `/v3/projects/4c5e5b3c09904d95bb215126b79de50f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:20.038808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 846/1694] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 474 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:20.038907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c5e5b3c09904d95bb215126b79de50f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:20.041308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:20.045713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:20.078272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af79164-01d2-4fa9-af38-7caf4c216cd0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:20.097781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57beb1f3-50cc-46cb-8e0c-e2ea114ad85b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:20.098688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57beb1f3-50cc-46cb-8e0c-e2ea114ad85b admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:20.099149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57beb1f3-50cc-46cb-8e0c-e2ea114ad85b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:20.099618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57beb1f3-50cc-46cb-8e0c-e2ea114ad85b admin admin] PATH_INFO: `/v3/projects/01199f8dbf6a441b9f2b6233a5ef1855/users/51bbd2b14c6b46d0ae846f973c088df2/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:20.110185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57beb1f3-50cc-46cb-8e0c-e2ea114ad85b admin admin] RBAC: Authorizing `identity:create_grant(project_id=01199f8dbf6a441b9f2b6233a5ef1855, user_id=51bbd2b14c6b46d0ae846f973c088df2, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:20.124161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57beb1f3-50cc-46cb-8e0c-e2ea114ad85b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:20.178446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 847/1695] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:20 2022] PUT /identity/v3/projects/01199f8dbf6a441b9f2b6233a5ef1855/users/51bbd2b14c6b46d0ae846f973c088df2/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:20.180926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c5e5b3c09904d95bb215126b79de50f', 'request_id': 'req-bbec4a4f-5287-4f91-9bd0-bca960c0e930'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.182793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:20.183307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:20.183969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:20.197210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c5e5b3c09904d95bb215126b79de50f', 'request_id': 'req-bbec4a4f-5287-4f91-9bd0-bca960c0e930'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.219324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c5e5b3c09904d95bb215126b79de50f', 'request_id': 'req-bbec4a4f-5287-4f91-9bd0-bca960c0e930'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.267530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] MFA Rules not processed for user `51bbd2b14c6b46d0ae846f973c088df2`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:20.269879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:20.285247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] The token cache is being invalidate because project 4c5e5b3c09904d95bb215126b79de50f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:20.286504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.288527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.293900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 849/1696] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:19 2022] DELETE /identity/v3/projects/4c5e5b3c09904d95bb215126b79de50f => generated 0 bytes in 619 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:10:20.301998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:20.314516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:20.427222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 848/1697] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:20 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:20.442104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:20.445050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:20.512902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:20.542538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:20.546131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbec4a4f-5287-4f91-9bd0-bca960c0e930 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:20.566946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:20.568739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:20.571753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:20.572411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] PATH_INFO: `/v3/users/05487dffe07d460a9620db919787350c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:20.578122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05487dffe07d460a9620db919787350c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:20.583262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:20.633927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d1254-997c-4197-ae96-bf4006495e45 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:20.679211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8eb9d7-b8e8-4ed5-9505-6e8ffa666c45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:20.680329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8eb9d7-b8e8-4ed5-9505-6e8ffa666c45 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:20.680805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8eb9d7-b8e8-4ed5-9505-6e8ffa666c45 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:20.681312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8eb9d7-b8e8-4ed5-9505-6e8ffa666c45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:20.712658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8eb9d7-b8e8-4ed5-9505-6e8ffa666c45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:20.717255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8eb9d7-b8e8-4ed5-9505-6e8ffa666c45 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:20.744382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05487dffe07d460a9620db919787350c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05487dffe07d460a9620db919787350c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.748849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05487dffe07d460a9620db919787350c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.767137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 849/1698] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:20 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:20.779466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05487dffe07d460a9620db919787350c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.824768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-02002b6f-e62d-4c3f-9f6f-128587a2bfe2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05487dffe07d460a9620db919787350c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:20.883735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 850/1699] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:20 2022] DELETE /identity/v3/users/05487dffe07d460a9620db919787350c => generated 0 bytes in 584 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:21.185982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c38bc85-7196-4687-a863-e32cb05ca3ce None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:21.186585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c38bc85-7196-4687-a863-e32cb05ca3ce None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:21.186585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c38bc85-7196-4687-a863-e32cb05ca3ce None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:21.188057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 850/1700] 10.222.0.26 () {54 vars in 912 bytes} [Sun Aug 7 01:10:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:10:21.195330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:21.195949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:21.195949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:21.245644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:21.247113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:21.338661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 851/1701] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:10:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:21.371928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:21.372562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:21.373103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:21.427274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] MFA Rules not processed for user `4ae9c7ef17b746fc8f87e3c22a241b76`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:21.429811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:21.465086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:21.467553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:21.521614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682ac49a-591c-4c1b-8c34-3a177707fcf5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:21.550327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:21.551960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:21.552486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:21.552965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:21.632750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:21.635731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:21.711092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 852/1702] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:10:21.818431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 851/1703] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 448 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:21.863894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:21.865710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:22.102046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:22.165712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068b630-63ad-4a7e-a7fa-3e1589e8c0d6 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:22.199936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52afb53f-7a85-49cf-8535-6a70e4bef426 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:22.199936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52afb53f-7a85-49cf-8535-6a70e4bef426 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:22.201129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52afb53f-7a85-49cf-8535-6a70e4bef426 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:22.201129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52afb53f-7a85-49cf-8535-6a70e4bef426 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:22.237932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52afb53f-7a85-49cf-8535-6a70e4bef426 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:22.242151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52afb53f-7a85-49cf-8535-6a70e4bef426 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:22.282951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 853/1704] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:22.609729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:22.612189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:22.872727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb1cc6-5c45-42b3-8f8b-7407ea61a1cd None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:22.884637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:22.885380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:22.885800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:22.886214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] PATH_INFO: `/v3/users/5f705757cfb2407e9e7d6487917bda7e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:22.887731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f705757cfb2407e9e7d6487917bda7e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:22.890979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:22.925428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:22.926357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:22.926878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:23.040581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] MFA Rules not processed for user `caafb1ccfd704e3ca65b6c4db04b05ef`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:23.042860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:23.160885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f705757cfb2407e9e7d6487917bda7e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.184286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f705757cfb2407e9e7d6487917bda7e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.198856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f705757cfb2407e9e7d6487917bda7e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.200195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f705757cfb2407e9e7d6487917bda7e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.201066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fedb8cd5-85dd-4f64-8fa0-7f3cd403696a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f705757cfb2407e9e7d6487917bda7e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.210002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:23.244002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 852/1705] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:22 2022] DELETE /identity/v3/users/5f705757cfb2407e9e7d6487917bda7e => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:23.248726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8dbb63-1165-4e85-81f1-9eb596673fd7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:23.249039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8dbb63-1165-4e85-81f1-9eb596673fd7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:23.249340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8dbb63-1165-4e85-81f1-9eb596673fd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:23.298842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0f8dbb63-1165-4e85-81f1-9eb596673fd7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:23.306245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0f8dbb63-1165-4e85-81f1-9eb596673fd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:23.372652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 854/1706] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 450 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:23.374628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:23.379626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:23.415815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564d91ba-070e-49fc-8a09-6bfc41906ef6 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:23.520277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 853/1707] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:23.523168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:23.523728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:23.524498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:23.563816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:23.564603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:23.564890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:23.565693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] PATH_INFO: `/v3/users/2a41d2a303b44f41b22cc6139aded533` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:23.568142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a41d2a303b44f41b22cc6139aded533)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:23.570310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:23.582134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] MFA Rules not processed for user `caafb1ccfd704e3ca65b6c4db04b05ef`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:23.588720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:23.684882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a41d2a303b44f41b22cc6139aded533'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.686536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a41d2a303b44f41b22cc6139aded533'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.688687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a41d2a303b44f41b22cc6139aded533'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.708943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a41d2a303b44f41b22cc6139aded533'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.724191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a41d2a303b44f41b22cc6139aded533'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:23.751033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:23.777157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 855/1708] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:23 2022] DELETE /identity/v3/users/2a41d2a303b44f41b22cc6139aded533 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:23.778014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:23.782667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:23.833740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 854/1709] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:23.834967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:23.841983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:24.103951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe4e5c1-7fb3-4938-80d4-0dfe02f8e921 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:24.105077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d56d46c-8fe7-4bd8-a09f-5bcb373911a6 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:24.112009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:24.116858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:24.117242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:24.117579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] PATH_INFO: `/v3/projects/d41642e6db5d49119944c297c84ddc38` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:24.127404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=d41642e6db5d49119944c297c84ddc38)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:24.130322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:24.247204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=2,keystone:SELECT=98 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:24.248943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd41642e6db5d49119944c297c84ddc38', 'request_id': 'req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:24.264834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:24.266947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:24.267457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:24.269219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:24.270133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd41642e6db5d49119944c297c84ddc38', 'request_id': 'req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:24.278100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:24.300258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd41642e6db5d49119944c297c84ddc38', 'request_id': 'req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:24.326498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] The token cache is being invalidate because project d41642e6db5d49119944c297c84ddc38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:24.328943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:24.329924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:24.330365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-54e5ca62-b3f5-43c4-89ea-91c84736c2fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:24.333219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 856/1710] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:23 2022] DELETE /identity/v3/projects/d41642e6db5d49119944c297c84ddc38 => generated 0 bytes in 556 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:24.339943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca191b9-9aad-438b-80ba-4af28d5efaeb None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:24.341388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca191b9-9aad-438b-80ba-4af28d5efaeb None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:24.341768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca191b9-9aad-438b-80ba-4af28d5efaeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:24.342057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:24.377518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 855/1711] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 543 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:24.396875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:24.406645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:24.407315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1ca191b9-9aad-438b-80ba-4af28d5efaeb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:24.409720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1ca191b9-9aad-438b-80ba-4af28d5efaeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:24.729008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 857/1712] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:24.730562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953112c4-dc0e-4686-85c2-d1f2f3c88920 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:24.737700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e6963-848b-40c4-ad35-63989ded22ca None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:24.740429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e6963-848b-40c4-ad35-63989ded22ca None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:24.741274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210e6963-848b-40c4-ad35-63989ded22ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:24.824920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-210e6963-848b-40c4-ad35-63989ded22ca None None] MFA Rules not processed for user `e65f0f1ba54f483abff788974b1256cb`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:24.827832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-210e6963-848b-40c4-ad35-63989ded22ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:24.904614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:24.953493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:24.954379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:24.954524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:24.954746 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:24.961520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.073429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 858/1713] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:25.076057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1824d5-f860-4c15-a15a-35733587e9a5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.080640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1824d5-f860-4c15-a15a-35733587e9a5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.081686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1824d5-f860-4c15-a15a-35733587e9a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.110493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:25.111367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a566d6af-d8ac-42f7-a6e7-4e816553ebc2 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:25.146017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4d1824d5-f860-4c15-a15a-35733587e9a5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:25.146671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 856/1714] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:24 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 751 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:25.148365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4201a73f-3ed5-4e26-85a3-5ebbe0f5f960 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.152789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4d1824d5-f860-4c15-a15a-35733587e9a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.152944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4201a73f-3ed5-4e26-85a3-5ebbe0f5f960 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.153222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4201a73f-3ed5-4e26-85a3-5ebbe0f5f960 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.160993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 857/1715] 10.222.0.26 () {54 vars in 961 bytes} [Sun Aug 7 01:10:25 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:10:25.162018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4201a73f-3ed5-4e26-85a3-5ebbe0f5f960 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:25.168829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4201a73f-3ed5-4e26-85a3-5ebbe0f5f960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.189750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 859/1716] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:25.197265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.197899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.198372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.257529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:25.261803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:25.264708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.361864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4201a73f-3ed5-4e26-85a3-5ebbe0f5f960 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:25.381093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc276882-bdf4-41a1-bd5e-d40219aaa849 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:25.381806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc276882-bdf4-41a1-bd5e-d40219aaa849 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.386340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc276882-bdf4-41a1-bd5e-d40219aaa849 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.386702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc276882-bdf4-41a1-bd5e-d40219aaa849 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.425222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc276882-bdf4-41a1-bd5e-d40219aaa849 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:25.427361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc276882-bdf4-41a1-bd5e-d40219aaa849 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:25.497891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 858/1717] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:25.506802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.516803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.521608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.568504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 860/1718] 10.222.0.26 () {58 vars in 1058 bytes} [Sun Aug 7 01:10:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:25.569706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:25.575902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.606756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:25.616740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.637936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86955d1-7d85-4146-900c-6d228cc48e3b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:25.658934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 859/1719] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:25.662812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:25.665050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:25.665877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.668567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.668905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] PATH_INFO: `/v3/projects/8aa8a1e6519c47f7b544945035640933` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.680067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8aa8a1e6519c47f7b544945035640933)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:25.684991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:25.711439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac93f420-da5a-40f9-a7d0-794f73b4467e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:25.726360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:25.727763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:25.729283 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:25.729875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:25.738750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:25.797532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:25.808576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:25.853347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8aa8a1e6519c47f7b544945035640933', 'request_id': 'req-2d24783f-d784-44b8-8ccb-16d7a9979526'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:25.859333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 860/1720] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:25.864728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:25.883462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8aa8a1e6519c47f7b544945035640933', 'request_id': 'req-2d24783f-d784-44b8-8ccb-16d7a9979526'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:25.904288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8aa8a1e6519c47f7b544945035640933', 'request_id': 'req-2d24783f-d784-44b8-8ccb-16d7a9979526'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:25.910955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930125fc-998b-45b8-9781-3ab4aafe2bc4 service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:25.957868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] The token cache is being invalidate because project 8aa8a1e6519c47f7b544945035640933 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:25.958369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:25.963488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:25.970049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 861/1721] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:25 2022] DELETE /identity/v3/projects/8aa8a1e6519c47f7b544945035640933 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:25.974418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:25.983063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:26.268796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:26.269493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:26.269623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:26.269829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:26.282733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:26.282733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:26.282733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:26.282733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:26.381831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d24783f-d784-44b8-8ccb-16d7a9979526 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:26.388231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 861/1722] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 528 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:26.401821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:26.407590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:26.408357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:26.408695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:26.408979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] PATH_INFO: `/v3/users/51bbd2b14c6b46d0ae846f973c088df2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:26.416399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=51bbd2b14c6b46d0ae846f973c088df2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:26.417354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:26.418766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:26.525257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0c494c-8505-4999-8d6f-cf6a118c6c72 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:26.565663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:26.567678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:26.571476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:26.572544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:26.577371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:26.593200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:26.688725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51bbd2b14c6b46d0ae846f973c088df2'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:26.693038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51bbd2b14c6b46d0ae846f973c088df2'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:26.695508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51bbd2b14c6b46d0ae846f973c088df2'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:26.712655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 862/1723] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:26 2022] POST /identity/v3/projects => generated 383 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:26.723991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:26.725214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51bbd2b14c6b46d0ae846f973c088df2'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:26.736325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:26.766708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1517db78-648d-4179-8132-859c868b5d7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51bbd2b14c6b46d0ae846f973c088df2'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:26.845398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 862/1724] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:25 2022] DELETE /identity/v3/users/51bbd2b14c6b46d0ae846f973c088df2 => generated 0 bytes in 871 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:26.973746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069377f-e129-4a3e-a0d8-97bf899c9d82 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:26.974036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069377f-e129-4a3e-a0d8-97bf899c9d82 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:26.974316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069377f-e129-4a3e-a0d8-97bf899c9d82 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:26.976564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 863/1725] 10.222.0.26 () {54 vars in 912 bytes} [Sun Aug 7 01:10:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:10:26.980707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5989ff-6819-409d-9861-2b7a207f4e68 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:26.989180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:26.989611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:26.989976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.051641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:27.051942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:27.053116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:27.195962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:27.200013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:27.200601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:27.201070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.204460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:27.218410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:27.263865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 864/1726] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:10:26 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:27.285883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:27.288102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:27.298498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 863/1727] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:26 2022] POST /identity/v3/users => generated 292 bytes in 582 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:10:27.309811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:27.321901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdbbbd6-ddc9-451a-a296-83dfbd118d7d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:27.330858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd5565b-5cf6-40ad-94cf-33ca82b8426f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:27.342022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:27.342699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:27.342983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:27.345521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.358491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:27.360154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:27.360519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:27.361116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.362226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:27.362844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:27.364099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:27.365281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:27.397850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 865/1728] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:27.413831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 864/1729] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:27 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:27.418056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:27.464714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf999c7-185b-4462-935f-26226e3f4496 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:27.480499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985a8461-2e57-430f-8116-c270284010ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:27.482239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a8461-2e57-430f-8116-c270284010ff admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:27.482572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a8461-2e57-430f-8116-c270284010ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:27.487971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a8461-2e57-430f-8116-c270284010ff admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.489007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985a8461-2e57-430f-8116-c270284010ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:27.491006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985a8461-2e57-430f-8116-c270284010ff admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:27.519888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 866/1730] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:27 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:27.526271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:27.547786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790b3302-294d-4c01-83c9-1dc04c35a437 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:27.583484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:27.586934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:27.586934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:27.586934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] PATH_INFO: `/v3/projects/53caf2071b4b43c7b893b3a42d373809/users/c8bee0bd65d34b24afb0e15013ee31cf/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.649007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53caf2071b4b43c7b893b3a42d373809, user_id=c8bee0bd65d34b24afb0e15013ee31cf, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:27.652595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:27.698452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 865/1731] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:27 2022] PUT /identity/v3/projects/53caf2071b4b43c7b893b3a42d373809/users/c8bee0bd65d34b24afb0e15013ee31cf/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:27.723313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:27.724009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:27.724618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:27.829783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] MFA Rules not processed for user `c8bee0bd65d34b24afb0e15013ee31cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:27.833400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:28.002487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:28.163290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 867/1732] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 441 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:28.170029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:28.371662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f844262-08c6-4603-9088-c2c5592e4e90 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:28.401970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:28.402808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:28.403097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:28.403386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:28.404317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:28.411526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:28.493547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 866/1733] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:28 2022] POST /identity/v3/projects => generated 383 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:10:28.497064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:28.518650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4b992-0ade-4149-8671-423f12fcb5ae None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:28.592423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:28.594390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:28.798397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:28.799342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:28.799728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:28.799728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:28.802845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:28.811270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:28.827777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56dd938-f88b-4dd3-995c-6f28a9c2c8cf admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:28.843598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:28.845527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:28.846092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:28.846630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] PATH_INFO: `/v3/projects/01199f8dbf6a441b9f2b6233a5ef1855` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:28.855858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=01199f8dbf6a441b9f2b6233a5ef1855)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:28.859200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:28.864420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 868/1734] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:28 2022] POST /identity/v3/users => generated 292 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:28.871606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:28.889349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:28.899135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a4053f-6115-457b-abf3-c03c55d0d1dd admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:28.918984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:28.919732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:28.920021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:28.921308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:28.921308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:28.922916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:28.976947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 869/1735] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:28 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:28.977900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:29.001986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a539ad-63b3-4536-bcf5-efc6a8585c9b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:29.015700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01199f8dbf6a441b9f2b6233a5ef1855', 'request_id': 'req-5331c832-11b4-4d09-a74c-d800526db8ff'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:29.036015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01199f8dbf6a441b9f2b6233a5ef1855', 'request_id': 'req-5331c832-11b4-4d09-a74c-d800526db8ff'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:29.072413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01199f8dbf6a441b9f2b6233a5ef1855', 'request_id': 'req-5331c832-11b4-4d09-a74c-d800526db8ff'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:29.089163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] The token cache is being invalidate because project 01199f8dbf6a441b9f2b6233a5ef1855 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:29.089500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:29.093204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5331c832-11b4-4d09-a74c-d800526db8ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:29.100352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 867/1736] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:28 2022] DELETE /identity/v3/projects/01199f8dbf6a441b9f2b6233a5ef1855 => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:29.106475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4a0d4-5308-4766-976a-471648cbd9b2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.107341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4a0d4-5308-4766-976a-471648cbd9b2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.108278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4a0d4-5308-4766-976a-471648cbd9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.140683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-20b4a0d4-5308-4766-976a-471648cbd9b2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:29.142717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-20b4a0d4-5308-4766-976a-471648cbd9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:29.150032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:29.151381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.151711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.152056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.152816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:29.155023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:29.159971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 868/1737] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:29.179919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.181400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.183664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.187238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 870/1738] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:28 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:29.192797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:29.195553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:29.255573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a263d5cf-d858-4f06-9053-3656267900ea admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:29.286280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:29.303600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:29.303835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:29.303835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.303835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.304392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] PATH_INFO: `/v3/projects/5add6891b6a24051bafdeb004558cf1b/users/6ad23cef9c33472899370113eee23d03/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.334666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 869/1739] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:29.339250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:29.373583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5add6891b6a24051bafdeb004558cf1b, user_id=6ad23cef9c33472899370113eee23d03, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:29.376993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd529a15-a822-4f3b-938b-4374bed9fef0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:29.379523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:29.391725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:29.393241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.393811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.394231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.396946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:29.398163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:29.402804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e89ef12-069a-4f7d-b06a-368be9aa4c77 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:29.415040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 871/1740] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:29 2022] PUT /identity/v3/projects/5add6891b6a24051bafdeb004558cf1b/users/6ad23cef9c33472899370113eee23d03/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:29.420518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:29.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:29.463506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 870/1741] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:29.466007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.466524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.467215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.571272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:29.609433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] MFA Rules not processed for user `6ad23cef9c33472899370113eee23d03`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:29.609433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:29.724032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef936f-62c7-4460-90ea-c69c1633fb45 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:29.761602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:29.763045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:29.763526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:29.764024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] PATH_INFO: `/v3/projects/cb5a2700217d4324bedb0014e43804db` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:29.766023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb5a2700217d4324bedb0014e43804db)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:29.768434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:29.989923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb5a2700217d4324bedb0014e43804db', 'request_id': 'req-daa31e2a-a036-40a7-b9b4-5b550e407e1f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:30.027280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb5a2700217d4324bedb0014e43804db', 'request_id': 'req-daa31e2a-a036-40a7-b9b4-5b550e407e1f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:30.046278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 871/1742] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 577 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:30.046278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:30.053876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb5a2700217d4324bedb0014e43804db', 'request_id': 'req-daa31e2a-a036-40a7-b9b4-5b550e407e1f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:30.072576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] The token cache is being invalidate because project cb5a2700217d4324bedb0014e43804db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:30.073062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:30.074210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:30.076985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 872/1743] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:29 2022] DELETE /identity/v3/projects/cb5a2700217d4324bedb0014e43804db => generated 0 bytes in 661 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:30.098637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:30.103456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:30.135440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20becee4-8741-42d6-98ef-894d8bea76f2 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:30.370082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa31e2a-a036-40a7-b9b4-5b550e407e1f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:30.400573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd7ef64-bc77-4f54-b2a5-493fcb4e8af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:30.402240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd7ef64-bc77-4f54-b2a5-493fcb4e8af3 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:30.402600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd7ef64-bc77-4f54-b2a5-493fcb4e8af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:30.405978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd7ef64-bc77-4f54-b2a5-493fcb4e8af3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:30.406711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd7ef64-bc77-4f54-b2a5-493fcb4e8af3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:30.421921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd7ef64-bc77-4f54-b2a5-493fcb4e8af3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:30.438089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2f9339-9bf2-4c60-80e8-7dfa25df774a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:30.438999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2f9339-9bf2-4c60-80e8-7dfa25df774a admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:30.439143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2f9339-9bf2-4c60-80e8-7dfa25df774a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:30.439403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2f9339-9bf2-4c60-80e8-7dfa25df774a admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:30.440012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2f9339-9bf2-4c60-80e8-7dfa25df774a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:30.443122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2f9339-9bf2-4c60-80e8-7dfa25df774a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:30.510251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:30.605857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 873/1744] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:30 2022] POST /identity/v3/projects => generated 387 bytes in 528 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:30.613570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:30.614123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:30.615941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:30.620138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 872/1745] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:30 2022] POST /identity/v3/projects => generated 405 bytes in 578 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:30.626932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:30.627273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:30.631763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:30.714189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] MFA Rules not processed for user `4ae9c7ef17b746fc8f87e3c22a241b76`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:30.715529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:30.775930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:30.785017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:30.969367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:31.103467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 873/1746] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 483 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:31.107070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.147548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 874/1747] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 541 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:31.149944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.152164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:31.172120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab9431e-46e2-425a-9334-e31b870710ec None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.180741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21db769-0f30-462e-a4a9-c5422cead48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.181828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21db769-0f30-462e-a4a9-c5422cead48a admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.182680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21db769-0f30-462e-a4a9-c5422cead48a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.188300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21db769-0f30-462e-a4a9-c5422cead48a admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.188300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21db769-0f30-462e-a4a9-c5422cead48a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.188300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21db769-0f30-462e-a4a9-c5422cead48a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.242270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 875/1748] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:31 2022] POST /identity/v3/users => generated 303 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:31.242270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.242270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.242887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.287521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:31.289639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:31.309012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 876/1749] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:31.309012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.343841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5f06c-a101-431f-8a64-d01dfefa63eb None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.371466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.372638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.373136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.373651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.374912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.383483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.407315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 877/1750] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:31 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:31.408732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.430822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad49aee-613d-4716-aa73-1cbe723ab487 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.443645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.445184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.446622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.447428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.449684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:31.450716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.454923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.529781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 878/1751] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:31.535638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.595248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bca2b09-7db7-40d7-af0a-4a7ba9c82d07 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.614870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fd23dd-7fad-4ab2-a543-2046b0ca5f76 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.617982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.619998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.620807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.621858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.623987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.629812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.631071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.631890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.632882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.635128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.637130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.645908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.685676 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 879/1752] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:31 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:31.690079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.728369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 874/1753] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:31 2022] POST /identity/v3/users => generated 300 bytes in 624 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:31.731426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.733153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00133ef-af95-41bd-933b-d1c633e226e5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.765130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60799c39-d446-4aca-847e-1a49880e997c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:31.775553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.777818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.778598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.779389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] PATH_INFO: `/v3/projects/29448aa989e7479e9c1b8502461c2fc6/users/82af6af0107d425ca27befea7779e4bd/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.782017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c57f561-ac38-415a-b0bf-765e566abbcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:31.784701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57f561-ac38-415a-b0bf-765e566abbcb admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:31.785596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57f561-ac38-415a-b0bf-765e566abbcb admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:31.785967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c57f561-ac38-415a-b0bf-765e566abbcb admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:31.786678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c57f561-ac38-415a-b0bf-765e566abbcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.805062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c57f561-ac38-415a-b0bf-765e566abbcb admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.848938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29448aa989e7479e9c1b8502461c2fc6, user_id=82af6af0107d425ca27befea7779e4bd, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:31.865831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:31.945575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 875/1754] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:31 2022] PUT /identity/v3/projects/29448aa989e7479e9c1b8502461c2fc6/users/82af6af0107d425ca27befea7779e4bd/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:31.947107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:31.992824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 880/1755] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:31 2022] POST /identity/v3/projects => generated 381 bytes in 307 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:32.002221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:32.002921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:32.003210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:32.075385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:32.077915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:32.254773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:32.288583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac057bea-f666-4761-94a2-ba7404b1a311 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:32.484367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 881/1756] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 491 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:32.485960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:32.543013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:32.546893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:32.547426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e91c79-9835-420c-86ad-bc35b0d6acfb None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:32.548543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:32.549116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:32.551941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:32.558545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:32.567379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:32.569923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:32.570416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:32.570894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:32.576709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:32.583246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:32.607896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 876/1757] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:31 2022] GET /identity/v3/roles => generated 1420 bytes in 662 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:32.618888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:32.663707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 882/1758] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:32 2022] POST /identity/v3/users => generated 291 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:32.670546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:32.676451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37493f3-d2d2-4359-9071-0a8cc0e8464f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:32.699702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:32.701323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:32.702060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:32.702764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] PATH_INFO: `/v3/projects/fcc2245f783443c683d7161c3b697b99/users/3b879904c09a49b48164dda0d6c0c2e1/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:32.734894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a49d94-53a3-436d-bc10-2215063b8e40 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:32.772000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:32.773203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:32.773698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:32.774190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:32.775340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:32.775561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcc2245f783443c683d7161c3b697b99, user_id=3b879904c09a49b48164dda0d6c0c2e1, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:32.777024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:32.783743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:32.800406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 883/1759] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:32.804743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:32.810154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:32.832224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 877/1760] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:32 2022] PUT /identity/v3/projects/fcc2245f783443c683d7161c3b697b99/users/3b879904c09a49b48164dda0d6c0c2e1/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:32.838878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:32.903909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:32.988086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7705ccc6-d38f-46bb-838c-9ed2dceed66f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:33.087511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502a4d18-30e4-4884-8c9f-cf8f5ad944b8 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:33.116691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:33.118535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.124258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.125223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.127828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:33.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:33.155818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:33.156168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.156494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.156877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.181583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:33.183631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:33.198444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 884/1761] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:33.199462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:33.251664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 878/1762] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 412 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:33.266085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:33.280314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d02e7a-0e5e-4c5e-888f-805bf0d9a5ab service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:33.329165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:33.330257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.335654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.336392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.339720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:33.348349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5c52b6-cb83-4547-9e3f-528a9afe5990 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:33.348554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:33.383842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932e2cd6-7b1c-4fc9-8b0b-bc2e9f9dfc7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:33.387775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932e2cd6-7b1c-4fc9-8b0b-bc2e9f9dfc7d admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.394670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932e2cd6-7b1c-4fc9-8b0b-bc2e9f9dfc7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.397295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932e2cd6-7b1c-4fc9-8b0b-bc2e9f9dfc7d admin admin] PATH_INFO: `/v3/projects/a927d9c21da04f3dbaa7e92a0ba224f5/users/bd4eba6f1523471e9ce05705e266b5ce/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.415506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 885/1763] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:33 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:33.417979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:33.469019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b71f7e0-6149-42c5-9488-a8d86b5fab1e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:33.482425 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edc83e8-c176-428f-816f-8931e921dfdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:33.483141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc83e8-c176-428f-816f-8931e921dfdf admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.483541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc83e8-c176-428f-816f-8931e921dfdf admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.483827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edc83e8-c176-428f-816f-8931e921dfdf admin admin] PATH_INFO: `/v3/projects/fcc2245f783443c683d7161c3b697b99/users/3b879904c09a49b48164dda0d6c0c2e1/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.490167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edc83e8-c176-428f-816f-8931e921dfdf admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcc2245f783443c683d7161c3b697b99, user_id=3b879904c09a49b48164dda0d6c0c2e1, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:33.492377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932e2cd6-7b1c-4fc9-8b0b-bc2e9f9dfc7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a927d9c21da04f3dbaa7e92a0ba224f5, user_id=bd4eba6f1523471e9ce05705e266b5ce, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:33.503023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edc83e8-c176-428f-816f-8931e921dfdf admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:33.540304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932e2cd6-7b1c-4fc9-8b0b-bc2e9f9dfc7d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:33.545127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 886/1764] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:33 2022] PUT /identity/v3/projects/fcc2245f783443c683d7161c3b697b99/users/3b879904c09a49b48164dda0d6c0c2e1/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:33.567369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.567948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.568551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.620837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 879/1765] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:33 2022] PUT /identity/v3/projects/a927d9c21da04f3dbaa7e92a0ba224f5/users/bd4eba6f1523471e9ce05705e266b5ce/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:33.654559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90b13d-b40a-4b05-82fb-dcf26466ede1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:33.655621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90b13d-b40a-4b05-82fb-dcf26466ede1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:33.656168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90b13d-b40a-4b05-82fb-dcf26466ede1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:33.754991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] MFA Rules not processed for user `3b879904c09a49b48164dda0d6c0c2e1`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:33.757165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:33.788524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9c90b13d-b40a-4b05-82fb-dcf26466ede1 None None] MFA Rules not processed for user `bd4eba6f1523471e9ce05705e266b5ce`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:33.791181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9c90b13d-b40a-4b05-82fb-dcf26466ede1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:33.952454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:34.195169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 880/1766] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:33 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 537 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:34.196072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:34.207904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:34.208604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:34.234215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 887/1767] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:33 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 670 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:34.235275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:34.463132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:34.500287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:34.501328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de80f954-751e-43e1-abd8-c4001c29ca2e None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:34.508681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:34.512768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:34.513321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:34.513613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:34.513891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:34.514631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:34.516975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:34.619183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 888/1768] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:34 2022] POST /identity/v3/projects => generated 377 bytes in 384 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:34.620913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:34.640019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:34.840591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 881/1769] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:10:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 651 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:34.847062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:34.907510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaa8af9-1475-435d-9da8-4fe8f3a5368c None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:34.939504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c4fd8a-6a05-4f06-8a8e-8832f0a06044 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:34.964352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:34.965033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:34.965417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:34.965782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:35.216229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:35.220428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:35.229817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:35.236908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:35.237211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:35.237573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:35.241065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 889/1770] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:34 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 621 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:35.245366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:35.247715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:35.269810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:35.273490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:35.370178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 882/1771] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:34 2022] POST /identity/v3/users => generated 295 bytes in 529 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:35.380896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:35.421643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8addcfaf-b7fe-4ed9-a55d-66965f5bf764 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:35.463050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:35.466264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:35.466888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:35.468662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:35.472259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:35.475172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:35.546893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 883/1772] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:35.553236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:35.605560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f5f5ac-c5f9-40c6-a4d5-5e3f40a85cc4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:35.634350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:35.638954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:35.639690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:35.640853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] PATH_INFO: `/v3/projects/02bb92324e7a49809bd3286ab1475537/users/a663946746674c25877058b22ec8576d/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:35.646723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfdeaf3-8353-4dfb-97ed-c3da02acf91b service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:35.661754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:35.663340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:35.663991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:35.664706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:35.682164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:35.741851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02bb92324e7a49809bd3286ab1475537, user_id=a663946746674c25877058b22ec8576d, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:35.752387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:35.797992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 884/1773] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:35 2022] PUT /identity/v3/projects/02bb92324e7a49809bd3286ab1475537/users/a663946746674c25877058b22ec8576d/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:35.799474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:35.806236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:35.915101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:35.925287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:36.029005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 890/1774] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:10:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 759 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:10:36.054789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78077d2f-9569-4e3b-b9ec-f12213f4d079 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:36.093802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269105c6-cf02-4c3d-a894-45b5e7cd430e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:36.098448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269105c6-cf02-4c3d-a894-45b5e7cd430e admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:36.102268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269105c6-cf02-4c3d-a894-45b5e7cd430e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:36.105045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269105c6-cf02-4c3d-a894-45b5e7cd430e admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:36.110406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269105c6-cf02-4c3d-a894-45b5e7cd430e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:36.116027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269105c6-cf02-4c3d-a894-45b5e7cd430e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:36.186219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 885/1775] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:10:36.193147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:36.232024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46f990-10bb-4c10-b902-7a63bee21730 service swift] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:36.265672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:36.266835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:36.267234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:36.267550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] PATH_INFO: `/v3/projects/02bb92324e7a49809bd3286ab1475537/users/a663946746674c25877058b22ec8576d/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:36.289779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=02bb92324e7a49809bd3286ab1475537, user_id=a663946746674c25877058b22ec8576d, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:36.293968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:36.367277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 891/1776] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:36 2022] PUT /identity/v3/projects/02bb92324e7a49809bd3286ab1475537/users/a663946746674c25877058b22ec8576d/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:36.383344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:36.384430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:36.385533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:36.516123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] MFA Rules not processed for user `a663946746674c25877058b22ec8576d`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:36.518430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:37.006640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 886/1777] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:36 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 625 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:37.095510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:37.539543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172940ae-71f5-4ddd-91fc-c8edca4242ec admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:37.558757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:37.559491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:37.559781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:37.560110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:37.625032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:37.635847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:37.716285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 892/1778] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:37 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 623 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:37.972798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:37.977639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:38.052890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:38.056624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926b0e8-e117-4e7d-9e1a-d9c015c29bcc None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:38.066467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:38.067962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:38.068457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:38.068830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e497bc3e-8d31-41b8-8253-ffcb7b5bb604 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:38.069005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:38.073687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:38.096011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:38.097083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:38.097538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:38.097931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:38.365403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:38.370669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:38.393819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:38.404090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 893/1779] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:38 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:38.418492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dedac52-cccb-48bd-9b60-ae7a9b3e5fee service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:38.474748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 887/1780] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:37 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:38.988912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:38.991551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:39.227201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:39.377343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b42cdc-5208-418b-a539-d1452f860412 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:39.403037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:39.404441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:39.405045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:39.405592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] PATH_INFO: `/v3/users/18725d3a879344bd9e79faed34f5724e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:39.408132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18725d3a879344bd9e79faed34f5724e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:39.419641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:39.640854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18725d3a879344bd9e79faed34f5724e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:39.644154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18725d3a879344bd9e79faed34f5724e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:39.654480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18725d3a879344bd9e79faed34f5724e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:39.681148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18725d3a879344bd9e79faed34f5724e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:39.738744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18725d3a879344bd9e79faed34f5724e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:39.823538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 894/1781] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:38 2022] DELETE /identity/v3/users/18725d3a879344bd9e79faed34f5724e => generated 0 bytes in 832 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:39.825533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf5723-a69d-43c6-8699-dae269827bf4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:39.826349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf5723-a69d-43c6-8699-dae269827bf4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:39.826349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacf5723-a69d-43c6-8699-dae269827bf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:39.896383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-eacf5723-a69d-43c6-8699-dae269827bf4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:39.898523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-eacf5723-a69d-43c6-8699-dae269827bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:40.037778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:40.328511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 888/1782] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 500 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:40.596954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:40.599835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:40.772761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:40.773101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:40.773425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:40.914187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b68327-3d9b-49d1-9369-fab2de671ed7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:40.926788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:40.943019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e35d21d-461f-473d-891b-a5f8c687e0e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:40.943684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e35d21d-461f-473d-891b-a5f8c687e0e4 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:40.944045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e35d21d-461f-473d-891b-a5f8c687e0e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:40.944755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e35d21d-461f-473d-891b-a5f8c687e0e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:40.949412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:40.981802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 889/1783] 10.222.0.17 () {58 vars in 1015 bytes} [Sun Aug 7 01:10:40 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:41.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e35d21d-461f-473d-891b-a5f8c687e0e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:41.022690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e35d21d-461f-473d-891b-a5f8c687e0e4 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:41.105698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 895/1784] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 510 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:41.108872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:41.112419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:41.174225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6035b63a-9102-475d-bb65-02b391755c68 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:41.201297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:41.207263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:41.210954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:41.211782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:41.256339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:41.269577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:41.344687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 890/1785] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:43.859812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:43.862526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:43.863902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:44.033974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] MFA Rules not processed for user `82af6af0107d425ca27befea7779e4bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:44.036780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:44.259791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:44.340944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 896/1786] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:43 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 483 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:44.352069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:44.355583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:44.426501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5393754-697b-472a-a15a-e960fa039d77 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:44.453573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:44.458394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:44.460174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:44.479026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:44.486826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:44.497566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:44.618821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 891/1787] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:44 2022] POST /identity/v3/projects => generated 403 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:44.623648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:44.649229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf9ee0-662f-4e42-b107-174cfec8d441 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:44.929097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:44.931722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:44.932564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:44.938054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:44.939801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:44.944964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:45.011427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 897/1788] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:44 2022] POST /identity/v3/users => generated 308 bytes in 388 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:45.014565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:45.063366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d55ad-3dc6-4bd0-afa2-0fe3b753e355 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:45.093345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:45.095467 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:45.096289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:45.096994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:45.099318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:45.106507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:45.141525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 892/1789] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:10:45.146586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:45.174026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6c8204-c732-489a-bfd0-bcf6df2f3bd9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:45.186047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:45.186986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:45.187330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:45.187799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] PATH_INFO: `/v3/projects/ef3d98cc8f9844b287cd588b85a14ee1/users/fe41f36233384d1a99ca7845b07da37d/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:45.238813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef3d98cc8f9844b287cd588b85a14ee1, user_id=fe41f36233384d1a99ca7845b07da37d, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:45.249324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:45.295137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 898/1790] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:45 2022] PUT /identity/v3/projects/ef3d98cc8f9844b287cd588b85a14ee1/users/fe41f36233384d1a99ca7845b07da37d/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:45.297520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:45.369828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21edea8-3cc7-44a2-a1a3-c201aa803dd2 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:45.730781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:45.731582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:45.731977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:45.732431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:45.733571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:45.735748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:45.762134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 893/1791] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:45 2022] GET /identity/v3/roles => generated 1420 bytes in 466 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:45.765848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:45.795628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca37b9-d4a4-4e60-b0f9-09e2a9ec2c9b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:45.811790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae70aa8-3bf2-42e5-ab09-dcfdd0bb5269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:45.813742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae70aa8-3bf2-42e5-ab09-dcfdd0bb5269 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:45.816708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae70aa8-3bf2-42e5-ab09-dcfdd0bb5269 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:45.819215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae70aa8-3bf2-42e5-ab09-dcfdd0bb5269 admin admin] PATH_INFO: `/v3/projects/ef3d98cc8f9844b287cd588b85a14ee1/users/fe41f36233384d1a99ca7845b07da37d/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:45.835721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae70aa8-3bf2-42e5-ab09-dcfdd0bb5269 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef3d98cc8f9844b287cd588b85a14ee1, user_id=fe41f36233384d1a99ca7845b07da37d, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:45.844156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae70aa8-3bf2-42e5-ab09-dcfdd0bb5269 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:45.899647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 899/1792] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:45 2022] PUT /identity/v3/projects/ef3d98cc8f9844b287cd588b85a14ee1/users/fe41f36233384d1a99ca7845b07da37d/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:46.072279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:46.075687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:46.109252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0324bd76-597f-4fa9-be75-4f353e5fe17f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:46.350857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:46.352070 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.352888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.353436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] PATH_INFO: `/v3/projects/cd7b4755285e47588730fc262e7ccbdc` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.355835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd7b4755285e47588730fc262e7ccbdc)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:46.359414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:46.377751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:46.395782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a7f16-2b9d-45a1-9423-d3b5e3a42a47 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.396446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a7f16-2b9d-45a1-9423-d3b5e3a42a47 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.396446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a7f16-2b9d-45a1-9423-d3b5e3a42a47 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.398108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 900/1793] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:10:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 07 01:10:46.415283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.415545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.415727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.440724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd7b4755285e47588730fc262e7ccbdc', 'request_id': 'req-95e3eadd-409d-487e-b4f6-b82d737bb499'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.447536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:46.449176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:46.450323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd7b4755285e47588730fc262e7ccbdc', 'request_id': 'req-95e3eadd-409d-487e-b4f6-b82d737bb499'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.463859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd7b4755285e47588730fc262e7ccbdc', 'request_id': 'req-95e3eadd-409d-487e-b4f6-b82d737bb499'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.492062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] The token cache is being invalidate because project cd7b4755285e47588730fc262e7ccbdc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:46.492437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.493174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-95e3eadd-409d-487e-b4f6-b82d737bb499 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.498223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 894/1794] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:46 2022] DELETE /identity/v3/projects/cd7b4755285e47588730fc262e7ccbdc => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:46.499926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2935d539-9468-4d6e-867d-58a804da2227 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.500406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2935d539-9468-4d6e-867d-58a804da2227 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.500731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2935d539-9468-4d6e-867d-58a804da2227 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.502429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 895/1795] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:10:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:10:46.503297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2935d539-9468-4d6e-867d-58a804da2227 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:46.505445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2935d539-9468-4d6e-867d-58a804da2227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:46.574200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 901/1796] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:10:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:46.626684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2935d539-9468-4d6e-867d-58a804da2227 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:46.632670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:46.634574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.634574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.634574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] PATH_INFO: `/v3/users/40cfa432fe8141f18c8a83a162f08a2c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.637007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40cfa432fe8141f18c8a83a162f08a2c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:46.639535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:46.682441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:46.684666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:46.707853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddd3419-5caa-4999-95cf-35e6eeeb1153 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:46.715580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ebf2d9-d837-4680-87d8-1705025e0643 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:46.716811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ebf2d9-d837-4680-87d8-1705025e0643 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.717426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ebf2d9-d837-4680-87d8-1705025e0643 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.717968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ebf2d9-d837-4680-87d8-1705025e0643 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.730136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40cfa432fe8141f18c8a83a162f08a2c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.730698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ebf2d9-d837-4680-87d8-1705025e0643 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:46.733212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ebf2d9-d837-4680-87d8-1705025e0643 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:46.746581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40cfa432fe8141f18c8a83a162f08a2c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.753502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 902/1797] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:10:46.760120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40cfa432fe8141f18c8a83a162f08a2c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.761233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40cfa432fe8141f18c8a83a162f08a2c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.762009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40cfa432fe8141f18c8a83a162f08a2c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:46.789982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 896/1798] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:46 2022] DELETE /identity/v3/users/40cfa432fe8141f18c8a83a162f08a2c => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:46.803142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:46.803420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:46.804577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:46.826580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:46.827858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:46.846415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:46.983488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 903/1799] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:10:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:47.335029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:47.338994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:47.387581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4263dd2-40d6-4ebe-8555-98f0ab4a79b2 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:47.409696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:47.409696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:47.409696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:47.409696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:47.465577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:47.474246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:47.551097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 897/1800] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:48.719693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:48.721282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:48.870363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf69dc0-d586-43f9-b28a-aaf1619ad971 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:48.892337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:48.893384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:48.893883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:48.894987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:48.898526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:49.261795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:49.274178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af27e7-3e8c-42af-82fa-0e38e556c897 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:49.309344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 904/1801] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:48 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 590 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:10:50.731162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:50.737578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:50.748699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:50.932380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cae8d-3952-48a7-a8ef-60ac65000196 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:50.949020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:50.949713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:50.949999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:50.950303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] PATH_INFO: `/v3/projects/07d5d8a81c374d10ba8ef88e23aa31f5` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:50.954463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=07d5d8a81c374d10ba8ef88e23aa31f5)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:50.961914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:51.201141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07d5d8a81c374d10ba8ef88e23aa31f5', 'request_id': 'req-35337027-d929-42b2-a0da-72a806c6ce3d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:51.224795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07d5d8a81c374d10ba8ef88e23aa31f5', 'request_id': 'req-35337027-d929-42b2-a0da-72a806c6ce3d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:51.250712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07d5d8a81c374d10ba8ef88e23aa31f5', 'request_id': 'req-35337027-d929-42b2-a0da-72a806c6ce3d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:51.335757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] The token cache is being invalidate because project 07d5d8a81c374d10ba8ef88e23aa31f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:51.336254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:51.341931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-35337027-d929-42b2-a0da-72a806c6ce3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:51.351807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 898/1802] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:50 2022] DELETE /identity/v3/projects/07d5d8a81c374d10ba8ef88e23aa31f5 => generated 0 bytes in 622 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:51.357743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b4e8a-13c4-4c16-9a2a-5f7e4525e150 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:51.358646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b4e8a-13c4-4c16-9a2a-5f7e4525e150 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:51.358646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b4e8a-13c4-4c16-9a2a-5f7e4525e150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:51.442692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-079b4e8a-13c4-4c16-9a2a-5f7e4525e150 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:51.448922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-079b4e8a-13c4-4c16-9a2a-5f7e4525e150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:51.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:51.670216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:51.670616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:51.776356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] MFA Rules not processed for user `fe41f36233384d1a99ca7845b07da37d`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:51.776356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:51.907215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 905/1803] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 550 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:51.921580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:51.922185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:51.922576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:51.959061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:51.994660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:52.005046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:52.062280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 906/1804] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:52.069422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:52.072073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:52.160786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145a7a04-4b7c-4c71-a584-1e1cd3e9de63 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:52.197034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 899/1805] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:51 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 528 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:52.198669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:52.209485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:52.214505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:52.222732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:52.223448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] PATH_INFO: `/v3/users/bd4eba6f1523471e9ce05705e266b5ce` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:52.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd4eba6f1523471e9ce05705e266b5ce)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:52.255517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:52.264136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06420e6-e7e6-4a7f-8d7f-0aeca31d0f99 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:52.302250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:52.303257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:52.303785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:52.322597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:52.322597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:52.322597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:52.322597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:52.367424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 900/1806] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:52.372370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:52.392949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:52.409437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37da559-ef9d-4665-95b2-5af9209bceae admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:52.434231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:52.442566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:52.442993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:52.443119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:52.446628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:52.449825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:52.456470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd4eba6f1523471e9ce05705e266b5ce'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:52.458005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd4eba6f1523471e9ce05705e266b5ce'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:52.459837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd4eba6f1523471e9ce05705e266b5ce'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:52.490977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd4eba6f1523471e9ce05705e266b5ce'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:52.515644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f79b196c-a4ee-4736-91f8-e1324f97f5ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd4eba6f1523471e9ce05705e266b5ce'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:52.572190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 901/1807] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:52 2022] POST /identity/v3/projects => generated 385 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:52.574411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:52.586071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:52.606678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 907/1808] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:52 2022] DELETE /identity/v3/users/bd4eba6f1523471e9ce05705e266b5ce => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:52.614786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:52.615141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:52.615428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:52.714160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:52.715895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:52.839061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7a46a9-f503-4424-836d-a2bc2c4958aa admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:53.133967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:53.134615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:53.134911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:53.135194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:53.135842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:53.138064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:53.192293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 908/1809] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 573 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:53.238957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 902/1810] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:52 2022] POST /identity/v3/users => generated 293 bytes in 665 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:53.246687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:53.249894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:53.256791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:53.305591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28032fd-cd6b-4942-8f7f-de07dd5a8b56 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:53.374367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:53.380734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:53.381281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:53.381747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:53.383227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:53.386098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:53.473914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 903/1811] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:53 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:53.475718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:53.524200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697464d-c033-4a49-9a1c-d5199c56c413 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:53.536383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4935d90-3ee0-4499-b658-75ab596377ac None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:53.545720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:53.546057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:53.546210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:53.546422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:53.547284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fd7966-3f6a-401c-b2ac-81e19da2b2d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:53.551432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fd7966-3f6a-401c-b2ac-81e19da2b2d1 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:53.551921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fd7966-3f6a-401c-b2ac-81e19da2b2d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:53.554203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fd7966-3f6a-401c-b2ac-81e19da2b2d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:53.554936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fd7966-3f6a-401c-b2ac-81e19da2b2d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:53.556694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fd7966-3f6a-401c-b2ac-81e19da2b2d1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:53.570965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:53.571668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:53.607561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 909/1812] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:53.620614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 904/1813] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:53 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:53.625753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:53.654537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b3d616-43cf-41ba-8702-a292029e624e service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:53.666745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:53.668101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:53.668756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:53.669458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] PATH_INFO: `/v3/projects/20beda824ff94cfe9f4731136e202697/users/1532e4ac99254639ba90be426379fc19/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:53.737614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:53.755628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20beda824ff94cfe9f4731136e202697, user_id=1532e4ac99254639ba90be426379fc19, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:53.772924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:53.832438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 910/1814] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:53 2022] PUT /identity/v3/projects/20beda824ff94cfe9f4731136e202697/users/1532e4ac99254639ba90be426379fc19/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:53.853907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:53.855028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:53.856038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:53.959526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] MFA Rules not processed for user `1532e4ac99254639ba90be426379fc19`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:53.964344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:54.264468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:54.350334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 905/1815] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 498 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:54.359664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:54.446149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:54.449688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:54.620452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7469072-3eb5-421f-b121-300cc8db0c01 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:54.638872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:54.640116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:54.641286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:54.641286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:54.644292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:54.645322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0225586c-5b9b-47e0-b257-9a2fbfe89c75 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:54.649070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:54.665490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:54.667093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:54.669257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:54.670181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] PATH_INFO: `/v3/users/c8bee0bd65d34b24afb0e15013ee31cf` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:54.676954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8bee0bd65d34b24afb0e15013ee31cf)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:54.685883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:54.727673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 911/1816] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:54 2022] POST /identity/v3/projects => generated 385 bytes in 363 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:54.728554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:54.851057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8bee0bd65d34b24afb0e15013ee31cf'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:54.871418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8bee0bd65d34b24afb0e15013ee31cf'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:54.895837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090ac442-636b-4751-9b21-aa4699152395 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:54.897674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8bee0bd65d34b24afb0e15013ee31cf'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:54.901243 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8bee0bd65d34b24afb0e15013ee31cf'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:54.903556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0321b58d-3f2c-4772-874e-719625c663d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8bee0bd65d34b24afb0e15013ee31cf'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:54.906001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:54.907202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:54.907711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:54.908440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:54.909536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:54.914828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:54.947801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 912/1817] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:10:54 2022] POST /identity/v3/users => generated 299 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:10:54.951355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 906/1818] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:54 2022] DELETE /identity/v3/users/c8bee0bd65d34b24afb0e15013ee31cf => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:54.953041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:54.954184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3429cd72-3550-450f-916e-f033099da30a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:54.954454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3429cd72-3550-450f-916e-f033099da30a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:54.954778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3429cd72-3550-450f-916e-f033099da30a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:54.955863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:54.981237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76beafc6-add0-4150-b865-a4f84edce3bf admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:54.982015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3429cd72-3550-450f-916e-f033099da30a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:54.983376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3429cd72-3550-450f-916e-f033099da30a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:55.132873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:55.133269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.133708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.134051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.135081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:55.136821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:55.141524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 907/1819] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:10:55.148475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:55.158010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3429cd72-3550-450f-916e-f033099da30a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:55.161460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 913/1820] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:54 2022] GET /identity/v3/roles => generated 1420 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:55.162240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3429cd72-3550-450f-916e-f033099da30a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:55.163050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:55.187949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:55.198645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b2e84f-5d40-46be-9c70-e61c4f1d966a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:55.210903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0260189-8c31-48df-a748-906ddf300dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:55.212469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0260189-8c31-48df-a748-906ddf300dd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.213011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0260189-8c31-48df-a748-906ddf300dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.213718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0260189-8c31-48df-a748-906ddf300dd0 admin admin] PATH_INFO: `/v3/projects/7279d4598243431385dfc8cd7dbf61c1/users/bc1b4c372e674de2a3d79b43e8ef1002/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.257259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0260189-8c31-48df-a748-906ddf300dd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7279d4598243431385dfc8cd7dbf61c1, user_id=bc1b4c372e674de2a3d79b43e8ef1002, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:55.263040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0260189-8c31-48df-a748-906ddf300dd0 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:55.308877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 914/1821] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:55 2022] PUT /identity/v3/projects/7279d4598243431385dfc8cd7dbf61c1/users/bc1b4c372e674de2a3d79b43e8ef1002/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:55.311462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9724a75-da4e-4e1c-a480-f30deed0c17d None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.313009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9724a75-da4e-4e1c-a480-f30deed0c17d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.313801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9724a75-da4e-4e1c-a480-f30deed0c17d None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.317004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 915/1822] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:10:55 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:10:55.318342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9724a75-da4e-4e1c-a480-f30deed0c17d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:55.377244 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3429cd72-3550-450f-916e-f033099da30a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:55.446015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9724a75-da4e-4e1c-a480-f30deed0c17d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:55.452919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:55.454058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.454567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.455070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.456422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:55.459012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:55.476185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 916/1823] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:10:55 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:10:55.477893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:55.479979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:55.510521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdac2170-7089-4604-8b5a-b20510451dac admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:55.519730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:55.521010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.521593 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.522085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] PATH_INFO: `/v3/projects/a927d9c21da04f3dbaa7e92a0ba224f5` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.524131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] RBAC: Authorizing `identity:delete_project(project_id=a927d9c21da04f3dbaa7e92a0ba224f5)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:55.529376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:55.545436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7283aa9-c943-4c59-a097-888ac71ddfb1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:55.548212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7283aa9-c943-4c59-a097-888ac71ddfb1 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.548693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7283aa9-c943-4c59-a097-888ac71ddfb1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.549096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7283aa9-c943-4c59-a097-888ac71ddfb1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.585913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7283aa9-c943-4c59-a097-888ac71ddfb1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:55.589600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7283aa9-c943-4c59-a097-888ac71ddfb1 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:55.612695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 908/1824] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 456 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:55.614856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.615144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.615488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.637183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a927d9c21da04f3dbaa7e92a0ba224f5', 'request_id': 'req-835d8523-53b9-472e-b756-1f38d68ee2da'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:55.668018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a927d9c21da04f3dbaa7e92a0ba224f5', 'request_id': 'req-835d8523-53b9-472e-b756-1f38d68ee2da'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:55.684414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a927d9c21da04f3dbaa7e92a0ba224f5', 'request_id': 'req-835d8523-53b9-472e-b756-1f38d68ee2da'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:55.685795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:55.690858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:55.712586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] The token cache is being invalidate because project a927d9c21da04f3dbaa7e92a0ba224f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:55.713131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:55.714447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:55.718647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 917/1825] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:55 2022] DELETE /identity/v3/projects/a927d9c21da04f3dbaa7e92a0ba224f5 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:55.719820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:55.723244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:55.881836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 909/1826] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:10:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:55.888917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:55.891507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:55.918611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:55.929508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835d8523-53b9-472e-b756-1f38d68ee2da admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:55.943752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:55.945149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:55.945812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:55.947268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] PATH_INFO: `/v3/projects/7279d4598243431385dfc8cd7dbf61c1/users/bc1b4c372e674de2a3d79b43e8ef1002/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:55.953683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7279d4598243431385dfc8cd7dbf61c1, user_id=bc1b4c372e674de2a3d79b43e8ef1002, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:55.959891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:55.982693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 918/1827] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:10:55 2022] PUT /identity/v3/projects/7279d4598243431385dfc8cd7dbf61c1/users/bc1b4c372e674de2a3d79b43e8ef1002/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:55.995189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:56.005240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ea6931-efa3-4769-b618-43878f566c33 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:56.009297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:56.014397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:56.170748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:56.172183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:56.173826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:56.174460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] PATH_INFO: `/v3/users/a663946746674c25877058b22ec8576d` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:56.177139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a663946746674c25877058b22ec8576d)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:56.181291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:56.205294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3465f53d-928a-425e-88ad-af8c5fac35d5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:56.214843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b1911f-af4a-4b00-851f-a31c4f21ee83 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:56.216643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1911f-af4a-4b00-851f-a31c4f21ee83 service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:56.217179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1911f-af4a-4b00-851f-a31c4f21ee83 service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:56.217687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1911f-af4a-4b00-851f-a31c4f21ee83 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:56.235876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b1911f-af4a-4b00-851f-a31c4f21ee83 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:56.241308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b1911f-af4a-4b00-851f-a31c4f21ee83 service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:56.283507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 919/1828] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:55 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:56.288002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:56.288542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:56.288677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a663946746674c25877058b22ec8576d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:56.289029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:56.323802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a663946746674c25877058b22ec8576d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:56.341995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a663946746674c25877058b22ec8576d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:56.342605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] MFA Rules not processed for user `bc1b4c372e674de2a3d79b43e8ef1002`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:56.344064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:56.354405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a663946746674c25877058b22ec8576d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:56.355574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a663946746674c25877058b22ec8576d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:56.403299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 910/1829] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:55 2022] DELETE /identity/v3/users/a663946746674c25877058b22ec8576d => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:56.506682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 920/1830] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:56 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:56.528202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:56.530453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:56.689337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8337350-f77a-4c61-8244-c17dda9d0d53 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:56.695821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31316ecf-bfc1-4766-bc33-af716d9ffa76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:56.696827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31316ecf-bfc1-4766-bc33-af716d9ffa76 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:56.697220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31316ecf-bfc1-4766-bc33-af716d9ffa76 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:56.697607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31316ecf-bfc1-4766-bc33-af716d9ffa76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:56.803653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:56.860598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31316ecf-bfc1-4766-bc33-af716d9ffa76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:56.864943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31316ecf-bfc1-4766-bc33-af716d9ffa76 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:56.888101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 911/1831] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:56 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:57.010207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:57.013240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:57.060985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fadb332-5fc4-4458-8917-82fa4d6209b5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:57.229235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:57.230454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:57.231631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:57.232051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] PATH_INFO: `/v3/users/1532e4ac99254639ba90be426379fc19` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:57.234446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] RBAC: Authorizing `identity:delete_user(user_id=1532e4ac99254639ba90be426379fc19)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:57.239071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:57.250150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:57.398337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1532e4ac99254639ba90be426379fc19'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:57.400395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1532e4ac99254639ba90be426379fc19'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:57.403335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1532e4ac99254639ba90be426379fc19'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:57.432175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1532e4ac99254639ba90be426379fc19'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:57.453499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1532e4ac99254639ba90be426379fc19'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:57.505871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 921/1832] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:57 2022] DELETE /identity/v3/users/1532e4ac99254639ba90be426379fc19 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:57.513047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:57.513541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:57.514281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:57.558311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:57.560753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:57.810817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 912/1833] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:57.886098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:57.887722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:58.027689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:58.030269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:58.050790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7dd753-3432-4cbb-b643-fa1c0782276b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:58.071901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98563f7d-3776-47b2-a1a9-234cb929447b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:58.073525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98563f7d-3776-47b2-a1a9-234cb929447b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:58.075779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98563f7d-3776-47b2-a1a9-234cb929447b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:58.077208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98563f7d-3776-47b2-a1a9-234cb929447b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:58.080225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8ca929-cfea-45c9-b001-76dd36002975 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:58.095118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:58.096797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:58.097949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:58.098453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] PATH_INFO: `/v3/projects/53caf2071b4b43c7b893b3a42d373809` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:58.102229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53caf2071b4b43c7b893b3a42d373809)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:58.116476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:58.137530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98563f7d-3776-47b2-a1a9-234cb929447b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:58.139745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98563f7d-3776-47b2-a1a9-234cb929447b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:58.194467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 922/1834] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:58.309796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53caf2071b4b43c7b893b3a42d373809', 'request_id': 'req-2d7808de-a230-405f-8ee1-1d21ad70a760'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.321330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e9f62-b1ec-4be3-b1a1-bb1eefded8ed None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:58.321843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e9f62-b1ec-4be3-b1a1-bb1eefded8ed None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:58.322149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e9f62-b1ec-4be3-b1a1-bb1eefded8ed None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:58.338872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53caf2071b4b43c7b893b3a42d373809', 'request_id': 'req-2d7808de-a230-405f-8ee1-1d21ad70a760'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.344788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 923/1835] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:10:58 2022] GET /identity => generated 270 bytes in 25 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:10:58.362836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53caf2071b4b43c7b893b3a42d373809', 'request_id': 'req-2d7808de-a230-405f-8ee1-1d21ad70a760'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.396950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] The token cache is being invalidate because project 53caf2071b4b43c7b893b3a42d373809 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:58.397493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.398750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.402076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 913/1836] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:58 2022] DELETE /identity/v3/projects/53caf2071b4b43c7b893b3a42d373809 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:58.407938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e9f62-b1ec-4be3-b1a1-bb1eefded8ed None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:58.411487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-096e9f62-b1ec-4be3-b1a1-bb1eefded8ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:58.455055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:58.463076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:58.683673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e9f62-b1ec-4be3-b1a1-bb1eefded8ed None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:58.703617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:58.705216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:58.705216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:58.705701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] PATH_INFO: `/v3/users/6ad23cef9c33472899370113eee23d03` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:58.713144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ad23cef9c33472899370113eee23d03)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:58.717310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:58.727669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7808de-a230-405f-8ee1-1d21ad70a760 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:58.742828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:58.746479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:58.747091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:58.748015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] PATH_INFO: `/v3/projects/02bb92324e7a49809bd3286ab1475537` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:58.751956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02bb92324e7a49809bd3286ab1475537)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:58.759513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:58.769812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:58.885199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:58.972126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ad23cef9c33472899370113eee23d03'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.972893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02bb92324e7a49809bd3286ab1475537', 'request_id': 'req-2f4bb709-7527-4759-9f75-6e1f02d5d040'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.973731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ad23cef9c33472899370113eee23d03'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.974873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ad23cef9c33472899370113eee23d03'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:58.986821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02bb92324e7a49809bd3286ab1475537', 'request_id': 'req-2f4bb709-7527-4759-9f75-6e1f02d5d040'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.002325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ad23cef9c33472899370113eee23d03'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.006147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02bb92324e7a49809bd3286ab1475537', 'request_id': 'req-2f4bb709-7527-4759-9f75-6e1f02d5d040'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.035583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ad23cef9c33472899370113eee23d03'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.042932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] The token cache is being invalidate because project 02bb92324e7a49809bd3286ab1475537 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:10:59.044513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.046052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2f4bb709-7527-4759-9f75-6e1f02d5d040 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.052749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 914/1837] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:10:58 2022] DELETE /identity/v3/projects/02bb92324e7a49809bd3286ab1475537 => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:10:59.055348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4660ea81-c718-48b3-bd8b-08642010fad9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:59.056350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4660ea81-c718-48b3-bd8b-08642010fad9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:59.058139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4660ea81-c718-48b3-bd8b-08642010fad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:59.105332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 924/1838] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:58 2022] DELETE /identity/v3/users/6ad23cef9c33472899370113eee23d03 => generated 0 bytes in 698 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:10:59.107084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:59.111298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.136052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4660ea81-c718-48b3-bd8b-08642010fad9 None None] MFA Rules not processed for user `06551751330a406f8f0003153245b6a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:59.157259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4660ea81-c718-48b3-bd8b-08642010fad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.440795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 915/1839] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:10:59 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:59.450559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b61939-613a-48f6-b5a0-cfbb602272c8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:59.451384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b61939-613a-48f6-b5a0-cfbb602272c8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:59.453996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b61939-613a-48f6-b5a0-cfbb602272c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:59.469034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe778c3-2e34-46fa-818f-bb2fb54595d4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:59.480936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:59.482369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:59.483015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:59.483607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] PATH_INFO: `/v3/users/82af6af0107d425ca27befea7779e4bd` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:59.486951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82af6af0107d425ca27befea7779e4bd)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:59.493480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:59.540723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e1b61939-613a-48f6-b5a0-cfbb602272c8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:59.543042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e1b61939-613a-48f6-b5a0-cfbb602272c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.563260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 916/1840] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:59.578228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:59.578880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:59.579447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:59.629793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:10:59.632560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.659671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 917/1841] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:10:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:10:59.659671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:59.660555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82af6af0107d425ca27befea7779e4bd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.661987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.662186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82af6af0107d425ca27befea7779e4bd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.665560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82af6af0107d425ca27befea7779e4bd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.688711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82af6af0107d425ca27befea7779e4bd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.698579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae6c213-4f2b-4295-9c96-4dc15f70c6dd None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:59.717941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82af6af0107d425ca27befea7779e4bd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:10:59.723456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:59.724722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:59.725868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:59.726363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:59.733640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.778108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 925/1842] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:10:59 2022] DELETE /identity/v3/users/82af6af0107d425ca27befea7779e4bd => generated 0 bytes in 672 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:10:59.780546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:10:59.786014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:10:59.798183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:59.802037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:59.850888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 918/1843] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:10:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:10:59.886393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:10:59.961428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e483c9e-557a-4437-b295-2698fa89f9b2 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:10:59.968349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:10:59.969353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:10:59.969848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:10:59.970342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:10:59.971271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:10:59.971840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:10:59.974313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:10:59.992865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 926/1844] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:10:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:10:59.999004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:00.031660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e400fdda-0707-46e7-b11d-61dc77d984ea service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:00.060569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:00.069345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.085987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.085987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.086941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:00.091406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:00.141672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 919/1845] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:10:59 2022] POST /identity/v3/projects => generated 367 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:00.146553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:00.182328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d65d45b-913d-47be-912b-f89412139cf4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:00.417984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346743d8-58f0-4b6c-bd50-8f8e1c86a1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:00.418981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346743d8-58f0-4b6c-bd50-8f8e1c86a1fb admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.419373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346743d8-58f0-4b6c-bd50-8f8e1c86a1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.419824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346743d8-58f0-4b6c-bd50-8f8e1c86a1fb admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.420909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346743d8-58f0-4b6c-bd50-8f8e1c86a1fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:00.426649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346743d8-58f0-4b6c-bd50-8f8e1c86a1fb admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:00.468875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 927/1846] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:00 2022] POST /identity/v3/users => generated 284 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:00.474392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:00.521580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3c2fe7-8d3b-44d5-8c88-5bd98cbc29b6 None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.521969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3c2fe7-8d3b-44d5-8c88-5bd98cbc29b6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.522256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3c2fe7-8d3b-44d5-8c88-5bd98cbc29b6 None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.526618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 928/1847] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:11:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:11:00.541524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebf3a6b-eba6-4915-b74e-59207e2dd7a6 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:00.556818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.559258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:00.560388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.561386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.561640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.561921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.562568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.562962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:00.567420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:00.607763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 920/1848] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:00 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:00.609136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:00.622386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcc1059-5133-4b27-ab69-60a96abf241f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:00.630142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:00.631038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.631512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.631979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.633191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:00.635173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:00.655000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:00.655740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 921/1849] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:00 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:00.657050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:00.659382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:00.665416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:00.699254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea49bd-30dd-4258-8e9d-bccbf9f3495c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:00.739401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:00.740619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:00.741309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:00.741782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] PATH_INFO: `/v3/projects/20beda824ff94cfe9f4731136e202697` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:00.744040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20beda824ff94cfe9f4731136e202697)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:00.748875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:00.773760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:00.884419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20beda824ff94cfe9f4731136e202697', 'request_id': 'req-8ec37706-be14-4353-988b-69878ac4d522'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:00.901904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20beda824ff94cfe9f4731136e202697', 'request_id': 'req-8ec37706-be14-4353-988b-69878ac4d522'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:00.917428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20beda824ff94cfe9f4731136e202697', 'request_id': 'req-8ec37706-be14-4353-988b-69878ac4d522'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:00.941404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] The token cache is being invalidate because project 20beda824ff94cfe9f4731136e202697 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:00.942106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:00.943139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:00.945086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 922/1850] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:00 2022] DELETE /identity/v3/projects/20beda824ff94cfe9f4731136e202697 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:00.950255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:00.952656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.053493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 929/1851] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:11:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 509 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:01.055622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:01.062098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.102974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec37706-be14-4353-988b-69878ac4d522 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:01.105781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b66f3de-2bcc-4f34-850d-152513a625d5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:01.110073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:01.111289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:01.111885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:01.113240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] PATH_INFO: `/v3/projects/becbfe34657e4a818c4154e488e6fc08/users/ec46dc58cb5644ca9b3ed3cfe38b8683/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:01.137038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=becbfe34657e4a818c4154e488e6fc08, user_id=ec46dc58cb5644ca9b3ed3cfe38b8683, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:01.140846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:01.141708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:01.142618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:01.143089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:01.143742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] PATH_INFO: `/v3/projects/5add6891b6a24051bafdeb004558cf1b` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:01.157688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5add6891b6a24051bafdeb004558cf1b)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:01.164688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:01.172508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 923/1852] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:00 2022] PUT /identity/v3/projects/becbfe34657e4a818c4154e488e6fc08/users/ec46dc58cb5644ca9b3ed3cfe38b8683/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:01.174841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:01.184129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.294352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:01.352465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5add6891b6a24051bafdeb004558cf1b', 'request_id': 'req-b2ec3ac5-9815-4005-915c-9b61ed16e428'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.359270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae016a66-3d61-4c41-b376-0b8c2a7769ea admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:01.370896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5add6891b6a24051bafdeb004558cf1b', 'request_id': 'req-b2ec3ac5-9815-4005-915c-9b61ed16e428'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.387483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5add6891b6a24051bafdeb004558cf1b', 'request_id': 'req-b2ec3ac5-9815-4005-915c-9b61ed16e428'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.411447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] The token cache is being invalidate because project 5add6891b6a24051bafdeb004558cf1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:01.411995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.417028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.421471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 930/1853] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:01 2022] DELETE /identity/v3/projects/5add6891b6a24051bafdeb004558cf1b => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:01.423365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:01.428724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.439180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:01.511591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:01.512884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:01.513430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:01.514165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] PATH_INFO: `/v3/projects/29448aa989e7479e9c1b8502461c2fc6` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:01.516716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29448aa989e7479e9c1b8502461c2fc6)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:01.519381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:01.591783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29448aa989e7479e9c1b8502461c2fc6', 'request_id': 'req-90020d54-3991-4350-b6e5-5e3c5c2f9f76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.598974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ec3ac5-9815-4005-915c-9b61ed16e428 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:01.611437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29448aa989e7479e9c1b8502461c2fc6', 'request_id': 'req-90020d54-3991-4350-b6e5-5e3c5c2f9f76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.637332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29448aa989e7479e9c1b8502461c2fc6', 'request_id': 'req-90020d54-3991-4350-b6e5-5e3c5c2f9f76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.666851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] The token cache is being invalidate because project 29448aa989e7479e9c1b8502461c2fc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:01.666851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.668848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-90020d54-3991-4350-b6e5-5e3c5c2f9f76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.680639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 924/1854] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:01 2022] DELETE /identity/v3/projects/29448aa989e7479e9c1b8502461c2fc6 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:01.684968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:01.685457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:01.685938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:01.725289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:01.726160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:01.726446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:01.726870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] PATH_INFO: `/v3/users/bc1b4c372e674de2a3d79b43e8ef1002` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:01.730085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc1b4c372e674de2a3d79b43e8ef1002)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:01.733427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:01.762852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] MFA Rules not processed for user `ec46dc58cb5644ca9b3ed3cfe38b8683`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:01.768722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.841973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc1b4c372e674de2a3d79b43e8ef1002'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.843170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc1b4c372e674de2a3d79b43e8ef1002'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.844077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc1b4c372e674de2a3d79b43e8ef1002'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.857190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc1b4c372e674de2a3d79b43e8ef1002'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.869466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc1b4c372e674de2a3d79b43e8ef1002'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:01.888186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:01.919832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 931/1855] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:01 2022] DELETE /identity/v3/users/bc1b4c372e674de2a3d79b43e8ef1002 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:01.922141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:01.927995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.932047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 925/1856] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:01 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:01.937354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:01.939373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:01.966144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d1b2f-b037-4d77-8658-03aae8001aab None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:02.073632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6766c1-60b6-44e5-bf19-7052c53dc907 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:02.081861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:02.082686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:02.083047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:02.083421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:02.084412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:02.085690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:02.101018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:02.102572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:02.109786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:02.111090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:02.111564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:02.112286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] PATH_INFO: `/v3/users/3b879904c09a49b48164dda0d6c0c2e1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:02.115154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b879904c09a49b48164dda0d6c0c2e1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:02.116765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 932/1857] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:02.117765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:02.119383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:02.120349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:02.136430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f31a0c-354a-4bf7-8cbc-b6940b984c03 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:02.144722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:02.145680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:02.146136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:02.146686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] PATH_INFO: `/v3/users/fe41f36233384d1a99ca7845b07da37d` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:02.148688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe41f36233384d1a99ca7845b07da37d)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:02.151033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:02.249505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b879904c09a49b48164dda0d6c0c2e1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.266312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b879904c09a49b48164dda0d6c0c2e1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.278336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe41f36233384d1a99ca7845b07da37d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.279462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe41f36233384d1a99ca7845b07da37d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.281298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe41f36233384d1a99ca7845b07da37d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.291570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b879904c09a49b48164dda0d6c0c2e1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.294236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b879904c09a49b48164dda0d6c0c2e1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.297699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b879904c09a49b48164dda0d6c0c2e1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.305280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe41f36233384d1a99ca7845b07da37d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.327024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe41f36233384d1a99ca7845b07da37d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:02.341368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 926/1858] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:01 2022] DELETE /identity/v3/users/3b879904c09a49b48164dda0d6c0c2e1 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:02.343712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:02.360188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:02.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 933/1859] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:02 2022] DELETE /identity/v3/users/fe41f36233384d1a99ca7845b07da37d => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:02.586022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db820e-8edb-4a4f-b05f-841b502ec9db admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:02.733156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:02.733879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:02.734176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:02.734488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:02.735261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:02.737935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:02.800633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 927/1860] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:02 2022] POST /identity/v3/projects => generated 365 bytes in 453 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:02.803168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:02.806847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:02.837708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffc81ff-3070-44c0-bfd8-d7958d8e7851 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:02.988825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:03.017179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.018194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.018680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.019597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.021355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.024024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:03.056905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:03.059482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 934/1861] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:02 2022] POST /identity/v3/users => generated 289 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:03.065103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:03.066478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:03.099095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7528c6-ff92-4475-bee4-5a4307e1d4ac admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:03.099354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:03.109758 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.111118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.112005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.112631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.115908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.117123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc46f8f-6800-4d0c-98bb-da050c36f62f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:03.119398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:03.132007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.133308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.135158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.136117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] PATH_INFO: `/v3/projects/7279d4598243431385dfc8cd7dbf61c1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.138987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] RBAC: Authorizing `identity:delete_project(project_id=7279d4598243431385dfc8cd7dbf61c1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.146111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:03.148360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 935/1862] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:03 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:03.149974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:03.173775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77976239-5e2f-4ee5-9edd-f077fdb9ae38 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:03.184000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.185413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.185900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.186825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] PATH_INFO: `/v3/projects/c5ecc3b3cbe543cabd6f05f292cbccf2/users/bcdb8177e84246c2bbe7b6391b399ad0/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.224451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5ecc3b3cbe543cabd6f05f292cbccf2, user_id=bcdb8177e84246c2bbe7b6391b399ad0, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.231057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:03.270229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 936/1863] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:03 2022] PUT /identity/v3/projects/c5ecc3b3cbe543cabd6f05f292cbccf2/users/bcdb8177e84246c2bbe7b6391b399ad0/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:03.280406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:03.390132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7279d4598243431385dfc8cd7dbf61c1', 'request_id': 'req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:03.419434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7279d4598243431385dfc8cd7dbf61c1', 'request_id': 'req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:03.449457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7279d4598243431385dfc8cd7dbf61c1', 'request_id': 'req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:03.491471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] The token cache is being invalidate because project 7279d4598243431385dfc8cd7dbf61c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:03.493057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:03.495039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba7fd3b8-a96f-4e7e-a214-5225b72e67df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:03.499812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 928/1864] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:03 2022] DELETE /identity/v3/projects/7279d4598243431385dfc8cd7dbf61c1 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:03.516324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.516831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.517426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.562064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43aa6c93-3cf6-4cbd-a1a5-e40e661a51c6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:03.590432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:03.598979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:03.797550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.798807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.799121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.799595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.800899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.803874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:03.848468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 929/1865] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:03.848468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:03.848885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 937/1866] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:03 2022] GET /identity/v3/roles => generated 1420 bytes in 571 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:03.851990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:03.855849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:03.858862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:03.876193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05942cf6-ca7e-444b-89da-ac0d3628b56e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:03.888447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faa94cd-3d68-41fc-8585-a20b640ff7ce admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:03.898758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.899893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:03.901284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.901916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.902536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] PATH_INFO: `/v3/projects/ef3d98cc8f9844b287cd588b85a14ee1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.903382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:03.903930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:03.904496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] PATH_INFO: `/v3/projects/fcc2245f783443c683d7161c3b697b99` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:03.907634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fcc2245f783443c683d7161c3b697b99)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.908395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef3d98cc8f9844b287cd588b85a14ee1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:03.911858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:03.912647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:04.036334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:04.064772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef3d98cc8f9844b287cd588b85a14ee1', 'request_id': 'req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.073415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fcc2245f783443c683d7161c3b697b99', 'request_id': 'req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.086554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef3d98cc8f9844b287cd588b85a14ee1', 'request_id': 'req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.088967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fcc2245f783443c683d7161c3b697b99', 'request_id': 'req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.111353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef3d98cc8f9844b287cd588b85a14ee1', 'request_id': 'req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.116337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fcc2245f783443c683d7161c3b697b99', 'request_id': 'req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.139805 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] The token cache is being invalidate because project ef3d98cc8f9844b287cd588b85a14ee1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:04.140288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.141009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.142956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 938/1867] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:03 2022] DELETE /identity/v3/projects/ef3d98cc8f9844b287cd588b85a14ee1 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:04.145448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:04.153773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:04.163041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] The token cache is being invalidate because project fcc2245f783443c683d7161c3b697b99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:04.163930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.164190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e57b66d4-2a99-46da-ba1e-7ef98d3f9813 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:04.172656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 930/1868] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:03 2022] DELETE /identity/v3/projects/fcc2245f783443c683d7161c3b697b99 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:04.172656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93576aad-9dd3-4e87-982c-b7d2f70d62e0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.172656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93576aad-9dd3-4e87-982c-b7d2f70d62e0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.172656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93576aad-9dd3-4e87-982c-b7d2f70d62e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.228136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-93576aad-9dd3-4e87-982c-b7d2f70d62e0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:04.230416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-93576aad-9dd3-4e87-982c-b7d2f70d62e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:04.241644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:04.380326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c5c6a-9460-4933-ac6e-5ba2275cbd65 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:04.445249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 931/1869] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:04.447850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b13aa-8912-4b74-bebd-cbbbf6dcf71a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.448416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b13aa-8912-4b74-bebd-cbbbf6dcf71a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.448888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300b13aa-8912-4b74-bebd-cbbbf6dcf71a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.497314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-300b13aa-8912-4b74-bebd-cbbbf6dcf71a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:04.499615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-300b13aa-8912-4b74-bebd-cbbbf6dcf71a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:04.524292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 932/1870] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:04.524292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.524292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.524292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.569769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:04.571906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:04.590724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 933/1871] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:04.592229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:04.609256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93bd54d-2ab9-4ff2-aee3-f94b9df8f9d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:04.610175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bd54d-2ab9-4ff2-aee3-f94b9df8f9d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.610667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bd54d-2ab9-4ff2-aee3-f94b9df8f9d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.611217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93bd54d-2ab9-4ff2-aee3-f94b9df8f9d6 admin admin] PATH_INFO: `/v3/projects/c5ecc3b3cbe543cabd6f05f292cbccf2/users/bcdb8177e84246c2bbe7b6391b399ad0/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.615582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93bd54d-2ab9-4ff2-aee3-f94b9df8f9d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5ecc3b3cbe543cabd6f05f292cbccf2, user_id=bcdb8177e84246c2bbe7b6391b399ad0, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:04.621544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93bd54d-2ab9-4ff2-aee3-f94b9df8f9d6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:04.640935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51652f8-51f0-4896-8960-0adad8f26957 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:04.664096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 939/1872] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:04 2022] PUT /identity/v3/projects/c5ecc3b3cbe543cabd6f05f292cbccf2/users/bcdb8177e84246c2bbe7b6391b399ad0/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:04.665773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:04.666940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f02265b-93fa-4be5-8ad2-ecb6c44c067a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.667119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.667497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.667813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f02265b-93fa-4be5-8ad2-ecb6c44c067a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.668103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.668423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f02265b-93fa-4be5-8ad2-ecb6c44c067a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.673920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:04.677423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:04.699325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0103f02c-18f0-40a7-96fe-bb50f63c7435 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:04.704714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5f02265b-93fa-4be5-8ad2-ecb6c44c067a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:04.707254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5f02265b-93fa-4be5-8ad2-ecb6c44c067a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:04.768876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 934/1873] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:04.774487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.774980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.775433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:04.854927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:04.857066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:04.955514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 940/1874] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:04.969575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:04.970653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:04.971132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.022266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:05.044569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] MFA Rules not processed for user `bcdb8177e84246c2bbe7b6391b399ad0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:05.048977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:05.078998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 935/1875] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:05.083701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.108681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dda2cb-057a-4bf7-aa1c-8f62cecadf5e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.122848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.124535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.126059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.126481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.129390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.131656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.213250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 936/1876] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:05 2022] POST /identity/v3/projects => generated 389 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:05.214613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.229115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 941/1877] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:04 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:05.230937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.255163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cecc303f-af01-4688-a4ab-43e069e9a013 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.394810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ae700-db28-4bbf-bc55-87ada1b4164d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.401639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.402449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.402919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.403296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.404156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:05.404590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.406594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.410721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:05.418258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.419210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.419683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.420162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.421247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:05.422633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.425523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.439824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 937/1878] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:05.442685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.454926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 942/1879] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:05.454926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.458738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:05.461563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c71cf-3f7f-4573-9a60-af9eed973b67 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.470107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.471536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.472992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.472992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.473847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.476147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.515337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 938/1880] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:05 2022] POST /identity/v3/users => generated 295 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:05.516823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.537597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbd325-51d0-45ec-8469-1b41eca89c94 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.545735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.546382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.546758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.547029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.548489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.551329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.591526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54abdb90-4f97-46d5-93b0-e7269654d6c4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.599321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.599619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 939/1881] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:05 2022] POST /identity/v3/projects => generated 357 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:05.599806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.599806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.600585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.600994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.683549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:05.710763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.712415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.725044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 943/1882] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:05 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:05.726033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.739298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ece212-c05e-4a41-b102-7f9489e0fb49 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.740588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f75c18d-7dfd-4c71-b004-ac980b34dfab service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.746726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.747487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.747827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.748108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.748831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.749188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.750532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.750830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.751047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.751678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.753958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.759248 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.765319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 944/1883] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:05 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:05.766011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.791332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fb7e50-b2a3-4985-9859-e416371177ab admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.801611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:05.802691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:05.803034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:05.803525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:05.804673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:05.808177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:05.824884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 940/1884] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:05 2022] POST /identity/v3/projects => generated 371 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:05.824884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.852780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 945/1885] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:05 2022] POST /identity/v3/users => generated 279 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:05.854612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:05.882561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d262e5fc-e762-478d-ab33-76007ae4beda admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:05.986113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cfc2eb-d4b7-4799-815b-0019ec79d81e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.005051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.007021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.007702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.009188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.010762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.014893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.056415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 941/1886] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:05 2022] GET /identity/v3/roles => generated 1420 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:06.058085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.079724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ca92aa-4479-430a-b546-b63c0be5bdb0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.094574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.095849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.096426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.097180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.098976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.102989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.119217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.120369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.120851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.121377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.122573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.126770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.136297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 942/1887] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:06 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:06.138429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.182854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 946/1888] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:05 2022] POST /identity/v3/users => generated 286 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:06.184761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.201045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c7b2ab-9fae-4bad-8e96-450acfbccbd9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.209529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c71f719-c7e7-4633-85c6-44fb9f0acd3e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.221342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.222416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.222901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.224856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.226504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.227999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.228275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.231346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.232077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] PATH_INFO: `/v3/projects/79c79ccedcba4af9b9319c128c9e1375/users/532e1bab7a48475d930c87afee05db40/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.256993 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 947/1889] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:06 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:06.259062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.286639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79c79ccedcba4af9b9319c128c9e1375, user_id=532e1bab7a48475d930c87afee05db40, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.287735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148eaea2-e604-41c3-abcf-78282aff5f5c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.293757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.298490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.301195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.301877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.302439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.308948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.314158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.338053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 943/1890] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:06 2022] PUT /identity/v3/projects/79c79ccedcba4af9b9319c128c9e1375/users/532e1bab7a48475d930c87afee05db40/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:06.340169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.340377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 948/1891] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:06 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:06.351875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.369006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:06.385981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237cbbaa-d6fa-43be-8a93-0d0baa891d82 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.672713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e963df2-1963-4f15-9f47-3bf43b06cd63 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:06.702719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.703804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.704354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.704863 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] PATH_INFO: `/v3/projects/64218ac0bd314ac2af6f50b34fec5abd/users/322c9ac0ebdd413faeccf0bf54083e29/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.759860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2aa7b6-e61c-4758-b162-6a36ec7dc583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:06.760900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2aa7b6-e61c-4758-b162-6a36ec7dc583 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.766897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2aa7b6-e61c-4758-b162-6a36ec7dc583 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.770346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2aa7b6-e61c-4758-b162-6a36ec7dc583 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.775273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2aa7b6-e61c-4758-b162-6a36ec7dc583 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.785827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2aa7b6-e61c-4758-b162-6a36ec7dc583 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.817455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=64218ac0bd314ac2af6f50b34fec5abd, user_id=322c9ac0ebdd413faeccf0bf54083e29, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:06.826199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:06.843833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 949/1892] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:06 2022] GET /identity/v3/roles => generated 1420 bytes in 489 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:06.845902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef6476-e84f-430e-b0df-c321242caac1 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:06.849161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef6476-e84f-430e-b0df-c321242caac1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:06.853797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ef6476-e84f-430e-b0df-c321242caac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:06.872310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 944/1893] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:06 2022] PUT /identity/v3/projects/64218ac0bd314ac2af6f50b34fec5abd/users/322c9ac0ebdd413faeccf0bf54083e29/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 533 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:06.876177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:06.955310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:06.974793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e6ef6476-e84f-430e-b0df-c321242caac1 None None] MFA Rules not processed for user `532e1bab7a48475d930c87afee05db40`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:06.977062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e6ef6476-e84f-430e-b0df-c321242caac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:07.229021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 950/1894] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:06 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 385 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:07.231508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc235865-c402-4acd-aecc-fcfb5de8cde2 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:07.232008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc235865-c402-4acd-aecc-fcfb5de8cde2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:07.233961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc235865-c402-4acd-aecc-fcfb5de8cde2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:07.251905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6037ce1a-26bc-4aad-997b-463d106d07cd admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:07.267391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:07.273402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:07.274757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:07.275316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] PATH_INFO: `/v3/projects/1c3de7a94e61473aa759f03c19119775/users/dfc9712739724c91bd9dd5fdaaebd642/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:07.299551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cc235865-c402-4acd-aecc-fcfb5de8cde2 None None] MFA Rules not processed for user `322c9ac0ebdd413faeccf0bf54083e29`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:07.307472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cc235865-c402-4acd-aecc-fcfb5de8cde2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:07.374155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c3de7a94e61473aa759f03c19119775, user_id=dfc9712739724c91bd9dd5fdaaebd642, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:07.409290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:07.446617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 945/1895] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:06 2022] PUT /identity/v3/projects/1c3de7a94e61473aa759f03c19119775/users/dfc9712739724c91bd9dd5fdaaebd642/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:07.448896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:07.682612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a64cc78-2eb2-4226-8196-80cd69e9bc02 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:07.698996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:07.701359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:07.701852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:07.702337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:07.703503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:07.707425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:07.713697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 951/1896] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:07 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 484 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:07.716304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:07.716819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:07.717323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:07.776145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:07.793390 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:07.801787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:07.818269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 946/1897] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:07 2022] POST /identity/v3/projects => generated 389 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:07.824778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:07.839982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 952/1898] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:07.842151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:07.877314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2afe0c-60fe-4e1e-95c5-54ae45e17f46 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.038308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327be427-cd83-4f46-b755-3caef73117cb admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.040650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.043668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.044269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.044942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.045900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.049351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.055334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.056291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.066537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.066537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.069197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.072675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.096402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 953/1899] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:07 2022] POST /identity/v3/users => generated 301 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:08.096402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:08.100832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:08.144994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:08.213982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 947/1900] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:07 2022] POST /identity/v3/projects => generated 357 bytes in 389 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:08.213982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:08.317327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4a978d-0569-461e-ae76-03eae3fb696c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.413228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57f49bb-1920-4d7f-9163-85c884326167 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.423769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.424919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.425969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.426476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.427674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.431457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.456137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 948/1901] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:08 2022] GET /identity/v3/roles => generated 1420 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:08.458779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:08.479872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef42cbb-acfa-4244-82af-40eb2fee1f4b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.494611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.501165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.502239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.502747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.503285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.504595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.508711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.511322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.512053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.512807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.548777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.553254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.589196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 954/1902] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:08.595283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:08.598210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 949/1903] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:08 2022] POST /identity/v3/users => generated 285 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:08.599381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:08.657998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c34bad8-ef44-4db9-b3a6-14c251766025 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.676315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83879edf-f948-4b54-99db-a61d0ee943d9 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.703980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.705370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.717271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.717836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.718479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:08.719822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:08.720010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.720339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:08.720831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] PATH_INFO: `/v3/projects/9f1e7ad1779f41c29ebe3e4f9c5742bc/users/a1f95a12e3d44f8ba6e0ea7277f987ea/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:08.732450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.788801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 950/1904] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:08 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:08.796945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f1e7ad1779f41c29ebe3e4f9c5742bc, user_id=a1f95a12e3d44f8ba6e0ea7277f987ea, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:08.803849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:08.811137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:08.861234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d6474-3d66-4ab0-ad0f-82f4f33c2f96 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:08.871142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 955/1905] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:08 2022] PUT /identity/v3/projects/9f1e7ad1779f41c29ebe3e4f9c5742bc/users/a1f95a12e3d44f8ba6e0ea7277f987ea/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:08.877284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:09.076475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:09.127640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:09.128944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:09.129418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:09.129884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] PATH_INFO: `/v3/projects/e5bd676288ef46ffb7840de3a3695734/users/b382e5f6f10441949fe0243c9a01ee08/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:09.133140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4314690-5e9a-482b-be28-bc69f94a1e6c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:09.151816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:09.153084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:09.153641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:09.154152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:09.156761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:09.161690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:09.176919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5bd676288ef46ffb7840de3a3695734, user_id=b382e5f6f10441949fe0243c9a01ee08, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:09.186071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:09.204535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 956/1906] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:08 2022] GET /identity/v3/roles => generated 1420 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:09.215606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:09.241224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03d3ddd-0e03-4498-8130-682861418c15 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:09.243621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 951/1907] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:08 2022] PUT /identity/v3/projects/e5bd676288ef46ffb7840de3a3695734/users/b382e5f6f10441949fe0243c9a01ee08/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:09.253150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:09.428186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:09.464851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea5fff2-fd1e-4a8e-be6c-2536611a5f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:09.466402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea5fff2-fd1e-4a8e-be6c-2536611a5f99 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:09.467658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea5fff2-fd1e-4a8e-be6c-2536611a5f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:09.468731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea5fff2-fd1e-4a8e-be6c-2536611a5f99 admin admin] PATH_INFO: `/v3/projects/9f1e7ad1779f41c29ebe3e4f9c5742bc/users/a1f95a12e3d44f8ba6e0ea7277f987ea/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:09.475894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea5fff2-fd1e-4a8e-be6c-2536611a5f99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f1e7ad1779f41c29ebe3e4f9c5742bc, user_id=a1f95a12e3d44f8ba6e0ea7277f987ea, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:09.483750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea5fff2-fd1e-4a8e-be6c-2536611a5f99 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:09.511480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 957/1908] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:09 2022] PUT /identity/v3/projects/9f1e7ad1779f41c29ebe3e4f9c5742bc/users/a1f95a12e3d44f8ba6e0ea7277f987ea/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:09.542825 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c618f0b-e7b8-4f58-9ff8-19a9efec182b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:09.554356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15fd48-f9e2-4c40-9384-35122625de64 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:09.554927 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15fd48-f9e2-4c40-9384-35122625de64 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:09.555429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15fd48-f9e2-4c40-9384-35122625de64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:09.725576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7c15fd48-f9e2-4c40-9384-35122625de64 None None] MFA Rules not processed for user `a1f95a12e3d44f8ba6e0ea7277f987ea`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:09.728032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7c15fd48-f9e2-4c40-9384-35122625de64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:09.832173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:09.832944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:09.833479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:09.833953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:09.839239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:09.841100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:09.869500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 952/1909] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:09 2022] GET /identity/v3/roles => generated 1420 bytes in 621 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:09.883125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:09.919143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc1ef5-9918-4f17-9cae-341dbf20d7ce admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:09.941783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:09.942640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:09.942925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:09.943228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] PATH_INFO: `/v3/projects/e5bd676288ef46ffb7840de3a3695734/users/b382e5f6f10441949fe0243c9a01ee08/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:09.953515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5bd676288ef46ffb7840de3a3695734, user_id=b382e5f6f10441949fe0243c9a01ee08, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:09.958674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:10.021457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 953/1910] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:09 2022] PUT /identity/v3/projects/e5bd676288ef46ffb7840de3a3695734/users/b382e5f6f10441949fe0243c9a01ee08/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:10.022466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:10.024655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 958/1911] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:09 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 474 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:10.032719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:10.040961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:10.040961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:10.041443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:10.119544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] MFA Rules not processed for user `b382e5f6f10441949fe0243c9a01ee08`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:10.130227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:10.209935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958c023-c1bd-4cd6-85c6-3cdcfd076b38 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:10.226645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:10.227750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:10.227924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:10.228094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] PATH_INFO: `/v3/users/ec46dc58cb5644ca9b3ed3cfe38b8683` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:10.233358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec46dc58cb5644ca9b3ed3cfe38b8683)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:10.235484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:10.238953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:10.356963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 959/1912] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:10 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:10.356963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:10.356963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:10.408274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec46dc58cb5644ca9b3ed3cfe38b8683'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:10.429039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec46dc58cb5644ca9b3ed3cfe38b8683'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:10.461115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec46dc58cb5644ca9b3ed3cfe38b8683'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:10.462272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec46dc58cb5644ca9b3ed3cfe38b8683'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:10.463203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec46dc58cb5644ca9b3ed3cfe38b8683'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:10.541359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 954/1913] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:10 2022] DELETE /identity/v3/users/ec46dc58cb5644ca9b3ed3cfe38b8683 => generated 0 bytes in 519 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:10.544810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:10.547130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:10.554095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fde2a53-f885-4a04-a779-c5375d9244ec None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:10.561928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:10.702813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:10.703749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:10.704206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:10.704522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:10.713106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0be110-d334-43d1-942f-77dd68129950 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:10.713813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:10.720705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:10.721703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:10.722207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:10.722669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:10.726567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:10.894966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:10.898180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ad8a7d-1a3d-4c38-9ca6-0b43c2f0e946 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:10.899202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:10.900909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac796ab2-bd3d-40bc-be34-4eec01089c93 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:10.925945 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 955/1914] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:10 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:10.929076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 960/1915] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:10 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 578 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:10.929552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:10.930242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:10.931898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:10.932311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:10.932669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:10.933179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:10.976129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:10.979132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:11.034414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] MFA Rules not processed for user `dfc9712739724c91bd9dd5fdaaebd642`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:11.041946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:11.105818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:11.202261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 961/1916] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:11.208822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:11.239407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 956/1917] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:10 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:11.240998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:11.247807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc3b157-bcca-451d-8632-923682d0891c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:11.251209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:11.255297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:11.256563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:11.257330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:11.257818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:11.262197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:11.399806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bd6ba7-e3f0-4426-a654-ad0e1a975fbf None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:11.405177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:11.406529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:11.406529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:11.407531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:11.420587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:11.422313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:11.431327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:11.436225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:11.439535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:11.441443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 957/1918] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:11.442732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:11.459646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:11.465822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 962/1919] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:11 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 263 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:11.513808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e331e942-407f-4ace-b1dd-4eb33f80e079 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:11.523841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:11.525153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:11.526140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:11.526692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:11.554625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:11.557646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:11.576257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 958/1920] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:11 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:12.440341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:12.444831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:12.476659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df82b289-f0b4-4dd5-aeac-d2c1098f1bba service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:12.485232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:12.486333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:12.486781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:12.488021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] PATH_INFO: `/v3/projects/becbfe34657e4a818c4154e488e6fc08` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:12.490401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] RBAC: Authorizing `identity:delete_project(project_id=becbfe34657e4a818c4154e488e6fc08)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:12.495549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:12.591631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:12.594654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:12.621804 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'becbfe34657e4a818c4154e488e6fc08', 'request_id': 'req-61eacfab-8e17-4c34-9c18-9ce3d942fbed'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:12.629719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'becbfe34657e4a818c4154e488e6fc08', 'request_id': 'req-61eacfab-8e17-4c34-9c18-9ce3d942fbed'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:12.642367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'becbfe34657e4a818c4154e488e6fc08', 'request_id': 'req-61eacfab-8e17-4c34-9c18-9ce3d942fbed'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:12.664298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce934b-d150-46d6-8ddb-ae1b3f5a74eb service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:12.664530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] The token cache is being invalidate because project becbfe34657e4a818c4154e488e6fc08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:12.664810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:12.666016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:12.677344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 963/1921] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:12 2022] DELETE /identity/v3/projects/becbfe34657e4a818c4154e488e6fc08 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:12.680150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:12.685640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:12.847277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:12.848442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:12.848908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:12.849373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] PATH_INFO: `/v3/users/532e1bab7a48475d930c87afee05db40` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:12.852327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] RBAC: Authorizing `identity:delete_user(user_id=532e1bab7a48475d930c87afee05db40)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:12.856399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:12.872785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eacfab-8e17-4c34-9c18-9ce3d942fbed admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:12.889052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:12.891261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:12.892600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:12.893698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] PATH_INFO: `/v3/users/bcdb8177e84246c2bbe7b6391b399ad0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:12.897224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcdb8177e84246c2bbe7b6391b399ad0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:12.905011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:12.977903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '532e1bab7a48475d930c87afee05db40'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:12.986529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:12.999379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '532e1bab7a48475d930c87afee05db40'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.043507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '532e1bab7a48475d930c87afee05db40'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.045627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcdb8177e84246c2bbe7b6391b399ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.046578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '532e1bab7a48475d930c87afee05db40'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.047632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcdb8177e84246c2bbe7b6391b399ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.048583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b232792e-c55d-4774-9bf9-560657d31734 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '532e1bab7a48475d930c87afee05db40'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.048779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcdb8177e84246c2bbe7b6391b399ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.069229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcdb8177e84246c2bbe7b6391b399ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.109459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 959/1922] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:12 2022] DELETE /identity/v3/users/532e1bab7a48475d930c87afee05db40 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:13.111186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcdb8177e84246c2bbe7b6391b399ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:13.115445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca5632-78ad-4939-9611-b0d3c202d9fb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:13.115738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca5632-78ad-4939-9611-b0d3c202d9fb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:13.116058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca5632-78ad-4939-9611-b0d3c202d9fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:13.141664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:13.154029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 964/1923] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:12 2022] DELETE /identity/v3/users/bcdb8177e84246c2bbe7b6391b399ad0 => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:13.166392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a5ca5632-78ad-4939-9611-b0d3c202d9fb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:13.167775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a5ca5632-78ad-4939-9611-b0d3c202d9fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:13.344196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 960/1924] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:13.383054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:13.386256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:13.508452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e847c5da-c294-4ec3-bc3e-27dca049adb6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:13.515160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:13.516125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:13.516856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:13.517182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:13.547440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:13.550664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:13.575936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 965/1925] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:13.741770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17f677-8048-47c1-b377-71f65edab82a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:13.744517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17f677-8048-47c1-b377-71f65edab82a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:13.744517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17f677-8048-47c1-b377-71f65edab82a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:13.794277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-cf17f677-8048-47c1-b377-71f65edab82a None None] MFA Rules not processed for user `dfc9712739724c91bd9dd5fdaaebd642`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:13.806208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cf17f677-8048-47c1-b377-71f65edab82a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:13.883083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:13.901079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:13.952358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 961/1926] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:13 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:13.966369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67c45e7-3a20-4fa6-80ff-fa9ea2307fb5 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:13.985300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:13.987653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:13.988115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:13.988635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] PATH_INFO: `/v3/projects/c5ecc3b3cbe543cabd6f05f292cbccf2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:14.001528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5ecc3b3cbe543cabd6f05f292cbccf2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:14.004108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:14.178929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5ecc3b3cbe543cabd6f05f292cbccf2', 'request_id': 'req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:14.211238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5ecc3b3cbe543cabd6f05f292cbccf2', 'request_id': 'req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:14.228633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5ecc3b3cbe543cabd6f05f292cbccf2', 'request_id': 'req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:14.260318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] The token cache is being invalidate because project c5ecc3b3cbe543cabd6f05f292cbccf2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:14.260771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:14.264504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebac1ff4-5a5a-43e7-8ff3-0e938257249e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:14.266842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 966/1927] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:13 2022] DELETE /identity/v3/projects/c5ecc3b3cbe543cabd6f05f292cbccf2 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:14.274620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:14.275112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:14.275484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:14.308024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:14.310614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:14.489218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 962/1928] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:14.496377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:14.496516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:14.496916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:14.536854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:14.539179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:14.562704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 967/1929] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:14.570873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:14.602995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae19af5-7b95-46e3-9862-269663c78880 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:14.627266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:14.629347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:14.629722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:14.630301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:14.631722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:14.633173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:14.637146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:14.661943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:14.686600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:14.693948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 963/1930] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:11:14.714292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beea5da-2094-4f25-a782-919f497a2b16 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:14.744771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:14.745875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:14.746384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:14.746886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:14.748091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:14.764717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:14.929032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 968/1931] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:14 2022] POST /identity/v3/projects => generated 383 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:14.930823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:14.954723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd48cde-f1a8-47ac-87f2-d5c4c47d848f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:15.088023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:15.089159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:15.089886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:15.090516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:15.092171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:15.097977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:15.147514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:15.148332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 964/1932] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:14 2022] POST /identity/v3/users => generated 292 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:15.171509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94de2c6f-e91e-4740-bc75-61af5a399710 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:15.186716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:15.187852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:15.188609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:15.189148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:15.190906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:15.194433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:15.248054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 969/1933] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:15 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:15.250011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:15.274117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96dab3e-a96c-4308-8988-a75c5cc1e05d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:15.280126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b71a97-0628-4cd5-912c-6522adee5369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:15.281204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b71a97-0628-4cd5-912c-6522adee5369 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:15.281653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b71a97-0628-4cd5-912c-6522adee5369 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:15.282335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b71a97-0628-4cd5-912c-6522adee5369 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:15.283397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b71a97-0628-4cd5-912c-6522adee5369 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:15.290829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b71a97-0628-4cd5-912c-6522adee5369 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:15.348109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 965/1934] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:15 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:15.354338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:15.384520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a5afc6-fd52-460d-a626-846b5c67cbf8 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:15.415181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:15.416409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:15.421201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:15.422214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] PATH_INFO: `/v3/projects/1ddd62c6275e46c494a288e0be07f230/users/cd8ea1aeef844ccc876c19b62bd9b260/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:15.448793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ddd62c6275e46c494a288e0be07f230, user_id=cd8ea1aeef844ccc876c19b62bd9b260, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:15.453444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:15.485710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 970/1935] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:15 2022] PUT /identity/v3/projects/1ddd62c6275e46c494a288e0be07f230/users/cd8ea1aeef844ccc876c19b62bd9b260/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:15.487671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:15.488169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:15.488692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:15.544956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] MFA Rules not processed for user `cd8ea1aeef844ccc876c19b62bd9b260`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:15.547671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:15.686086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:15.692015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 966/1936] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:15.729501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:15.831986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4723fccb-26eb-4db1-b6f0-1a97c461f64c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:15.842810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:15.844436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:15.845188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:15.845801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:15.848724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:15.852179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:15.982940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:15.995639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 971/1937] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:15 2022] POST /identity/v3/projects => generated 381 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:16.013828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686b1ada-be6b-4a68-a0a0-26f01070e633 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:16.252927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:16.254743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:16.255231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:16.255706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:16.256908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:16.262728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:16.320579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:16.349094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:16.351982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 967/1938] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:15 2022] POST /identity/v3/users => generated 297 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:16.369898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d6eaf-2f7f-4ab7-8912-de5ad16cd1f9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:16.435440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:16.441325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:16.443363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:16.443820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:16.444263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:16.448408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:16.451152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:16.456515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:16.527889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2707ecda-26db-4313-9189-db2c30dcd39e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:16.529649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 972/1939] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:16.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:16.542732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:16.544071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:16.544418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:16.544736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] PATH_INFO: `/v3/projects/79c79ccedcba4af9b9319c128c9e1375` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:16.547655 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79c79ccedcba4af9b9319c128c9e1375)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:16.553340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:16.558249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963b6bea-4691-4b89-862f-4b66841a1271 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:16.576495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:16.578513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:16.578724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:16.578877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] PATH_INFO: `/v3/projects/1a1531dd993449e2b625964521acf7c0/users/343e70635e7749caadfe6b876283ddc3/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:16.629387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a1531dd993449e2b625964521acf7c0, user_id=343e70635e7749caadfe6b876283ddc3, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:16.633582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:16.663534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 973/1940] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:16 2022] PUT /identity/v3/projects/1a1531dd993449e2b625964521acf7c0/users/343e70635e7749caadfe6b876283ddc3/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:16.669290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:16.701930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79c79ccedcba4af9b9319c128c9e1375', 'request_id': 'req-7b78cea8-ad37-4809-8a66-67dd844f7b13'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:16.712494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79c79ccedcba4af9b9319c128c9e1375', 'request_id': 'req-7b78cea8-ad37-4809-8a66-67dd844f7b13'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:16.727980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79c79ccedcba4af9b9319c128c9e1375', 'request_id': 'req-7b78cea8-ad37-4809-8a66-67dd844f7b13'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:16.758164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] The token cache is being invalidate because project 79c79ccedcba4af9b9319c128c9e1375 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:16.760509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:16.763814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:16.769335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 968/1941] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:16 2022] DELETE /identity/v3/projects/79c79ccedcba4af9b9319c128c9e1375 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:16.775817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:16.779395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:16.805404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270f7068-3c5e-4ab9-9ba5-86178818fcd3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:16.943695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b78cea8-ad37-4809-8a66-67dd844f7b13 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:16.967619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:16.968352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:16.968633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:16.968980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] PATH_INFO: `/v3/users/a1f95a12e3d44f8ba6e0ea7277f987ea` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:16.972992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1f95a12e3d44f8ba6e0ea7277f987ea)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:16.975138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:16.980801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:16.983602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:16.984387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:16.985217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:16.986960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:16.991876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:17.022545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 974/1942] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:17.028028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:17.033101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:17.071652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c412d5-c022-4caa-b1c4-cb2ca1ce5c8e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:17.077553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80f65d1-a247-4228-b28f-abdba4386c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:17.078472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f65d1-a247-4228-b28f-abdba4386c2c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:17.078857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f65d1-a247-4228-b28f-abdba4386c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:17.079254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80f65d1-a247-4228-b28f-abdba4386c2c admin admin] PATH_INFO: `/v3/projects/1a1531dd993449e2b625964521acf7c0/users/343e70635e7749caadfe6b876283ddc3/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:17.083349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80f65d1-a247-4228-b28f-abdba4386c2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a1531dd993449e2b625964521acf7c0, user_id=343e70635e7749caadfe6b876283ddc3, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:17.087964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80f65d1-a247-4228-b28f-abdba4386c2c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:17.102048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:17.113868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 975/1943] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:17 2022] PUT /identity/v3/projects/1a1531dd993449e2b625964521acf7c0/users/343e70635e7749caadfe6b876283ddc3/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:17.129122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:17.129653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:17.130098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:17.201554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1f95a12e3d44f8ba6e0ea7277f987ea'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:17.209796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] MFA Rules not processed for user `343e70635e7749caadfe6b876283ddc3`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:17.211923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:17.224952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1f95a12e3d44f8ba6e0ea7277f987ea'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:17.276405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1f95a12e3d44f8ba6e0ea7277f987ea'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:17.281062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1f95a12e3d44f8ba6e0ea7277f987ea'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:17.284604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1f95a12e3d44f8ba6e0ea7277f987ea'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:17.339546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:17.362309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 969/1944] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:16 2022] DELETE /identity/v3/users/a1f95a12e3d44f8ba6e0ea7277f987ea => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:17.460730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 976/1945] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:17 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:17.483381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:17.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:17.753551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054bfe7b-525b-4ddc-97dd-65d8bfc7a032 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:17.761127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:17.762448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:17.763089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:17.763728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:18.026944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:18.030520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:18.056096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 970/1946] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:17 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 575 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:18.260555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:18.381716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c263b4-abf8-4f3c-adc4-23c10ba53135 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:18.386876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:18.387525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:18.387831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:18.388106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:18.390402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:18.598761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:18.601962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65854c1-ed5a-41f2-bb06-3aa6d3d075aa service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:18.623833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 977/1947] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:18.754596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:18.758403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:19.153789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0fdcb9-415b-41d2-bc7a-6b13a1cf7def service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:19.174564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:19.175712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:19.179302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:19.179797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] PATH_INFO: `/v3/projects/9f1e7ad1779f41c29ebe3e4f9c5742bc` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:19.186660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f1e7ad1779f41c29ebe3e4f9c5742bc)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:19.198996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:19.411844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f1e7ad1779f41c29ebe3e4f9c5742bc', 'request_id': 'req-dca3b99f-3d5f-458b-9120-1c5984a02ae2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:19.416632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:19.430935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f1e7ad1779f41c29ebe3e4f9c5742bc', 'request_id': 'req-dca3b99f-3d5f-458b-9120-1c5984a02ae2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:19.474517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f1e7ad1779f41c29ebe3e4f9c5742bc', 'request_id': 'req-dca3b99f-3d5f-458b-9120-1c5984a02ae2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:19.533395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] The token cache is being invalidate because project 9f1e7ad1779f41c29ebe3e4f9c5742bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:19.533943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:19.536947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:19.549612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 971/1948] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:18 2022] DELETE /identity/v3/projects/9f1e7ad1779f41c29ebe3e4f9c5742bc => generated 0 bytes in 797 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:19.559147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf5861-f127-4828-b471-6bda49350c44 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:19.559958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf5861-f127-4828-b471-6bda49350c44 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:19.560095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf5861-f127-4828-b471-6bda49350c44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:19.615821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-96bf5861-f127-4828-b471-6bda49350c44 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:19.621215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-96bf5861-f127-4828-b471-6bda49350c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:19.796444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:19.840271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:19.843528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:19.908498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 978/1949] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:19.915929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:19.916467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:19.917041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:19.975356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:19.981653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:20.057104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 979/1950] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:20.060166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:20.106172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4277c6aa-47a4-4273-90e7-41f97a8eeb42 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:20.118729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:20.120324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:20.122838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:20.123754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:20.126763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:20.127418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:20.134324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:20.163458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 980/1951] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:20.177933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:20.188824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3b99f-3d5f-458b-9120-1c5984a02ae2 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:20.196598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:20.197719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:20.198229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:20.198729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] PATH_INFO: `/v3/users/cd8ea1aeef844ccc876c19b62bd9b260` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:20.200784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd8ea1aeef844ccc876c19b62bd9b260)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:20.207755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:20.261569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d060b5c3-6950-4aee-ba16-92d88245a072 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:20.293000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:20.293670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:20.293960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:20.300620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:20.301372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:20.320805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd8ea1aeef844ccc876c19b62bd9b260'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:20.333819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd8ea1aeef844ccc876c19b62bd9b260'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:20.336724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:20.352493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd8ea1aeef844ccc876c19b62bd9b260'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:20.353777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd8ea1aeef844ccc876c19b62bd9b260'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:20.354656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0049e74f-4804-4793-b599-0a20534c867a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd8ea1aeef844ccc876c19b62bd9b260'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:20.382290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 972/1952] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:19 2022] DELETE /identity/v3/users/cd8ea1aeef844ccc876c19b62bd9b260 => generated 0 bytes in 543 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:20.388405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:20.388932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:20.389610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:20.421880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:20.426824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:20.495690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 981/1953] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:20 2022] POST /identity/v3/projects => generated 387 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:20.501709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:20.506367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:20.603721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:20.638833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 973/1954] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:20.698197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:20.699712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:20.742897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99edf2b2-1c63-423d-8fbc-8e073bd10387 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:20.756281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:20.759340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:20.760584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:20.760584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:20.762720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:20.770328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:20.827393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 982/1955] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:20 2022] POST /identity/v3/users => generated 294 bytes in 327 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:20.828389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:20.868562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0647ec6-f3d6-454f-8a75-fc3ebd9cf0e0 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:20.887856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:20.888528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:20.889384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b77ac85-15f4-4ee4-982b-8ca1505354d4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:20.894921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:20.895222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:20.898033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:20.911632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:20.960485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 983/1956] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:20 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:20.967553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:20.998299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac25cbb7-e5f3-4d47-bdc1-d9b4b2e55dde admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:21.031087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:21.032023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:21.032614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:21.033059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:21.034799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:21.037945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:21.066593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3b5041-8d1e-4331-99ed-0cc4c86f8a6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:21.067581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3b5041-8d1e-4331-99ed-0cc4c86f8a6a service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:21.067942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3b5041-8d1e-4331-99ed-0cc4c86f8a6a service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:21.068220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3b5041-8d1e-4331-99ed-0cc4c86f8a6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:21.069358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 984/1957] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:20 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:21.071097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:21.100218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057fea45-5486-4d50-adb6-ca3b2f4093f2 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:21.109618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3b5041-8d1e-4331-99ed-0cc4c86f8a6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:21.113349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3b5041-8d1e-4331-99ed-0cc4c86f8a6a service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:21.116561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:21.118295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:21.118701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:21.121146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] PATH_INFO: `/v3/projects/49321854a38d457780ed1bac7afce5a5/users/b308b1d021444ab1bacda522c3e277ef/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:21.139522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 974/1958] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:11:21.164735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49321854a38d457780ed1bac7afce5a5, user_id=b308b1d021444ab1bacda522c3e277ef, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:21.168080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:21.219454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 985/1959] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:21 2022] PUT /identity/v3/projects/49321854a38d457780ed1bac7afce5a5/users/b308b1d021444ab1bacda522c3e277ef/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:21.224787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:21.225275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:21.225675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:21.307360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] MFA Rules not processed for user `b308b1d021444ab1bacda522c3e277ef`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:21.312180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:21.512644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 975/1960] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:21 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:21.517669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:21.622066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:21.702672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654c3511-445f-43bb-ab3a-bbb772777ff1 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:21.722011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:21.725558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:21.725558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:21.725812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:21.726487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:21.731112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:21.830782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 986/1961] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:21 2022] POST /identity/v3/projects => generated 383 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:21.837063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:21.869791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a7d809-be79-4a85-ae01-b0e27eadb246 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:21.983502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:22.013459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.014513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.015075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.015533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.016584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.019148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.068216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 976/1962] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:21 2022] POST /identity/v3/users => generated 298 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:22.069596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:22.087838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac75c75-2878-495f-a51e-95fa3b534a63 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:22.098290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.099393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.099908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.100518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.103535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.107012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.129865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 987/1963] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:22 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:22.135141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:22.152588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036d67d8-3652-4c00-82da-e9232d4ef02d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:22.161403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.162501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.163053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.163468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] PATH_INFO: `/v3/projects/c6333b27309b4ad0b9429a8117996f9c/users/92c5b7cb92954678a5e66f208b9ee98a/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.196697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6333b27309b4ad0b9429a8117996f9c, user_id=92c5b7cb92954678a5e66f208b9ee98a, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.201293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.240095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 977/1964] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:22 2022] PUT /identity/v3/projects/c6333b27309b4ad0b9429a8117996f9c/users/92c5b7cb92954678a5e66f208b9ee98a/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:22.242501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:22.293303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483c0748-85cf-4eda-ba07-e5f62c2a371e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:22.393331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952cca4a-0354-4457-b833-2b297abf1670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.394018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952cca4a-0354-4457-b833-2b297abf1670 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.394295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952cca4a-0354-4457-b833-2b297abf1670 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.394568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952cca4a-0354-4457-b833-2b297abf1670 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.395232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952cca4a-0354-4457-b833-2b297abf1670 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.397744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952cca4a-0354-4457-b833-2b297abf1670 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.415574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 988/1965] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:22 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:22.419353 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:22.439750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dce87a-a510-4be2-bc24-2b8216ae6610 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:22.449405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.450567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.451063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.451572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] PATH_INFO: `/v3/projects/c6333b27309b4ad0b9429a8117996f9c/users/92c5b7cb92954678a5e66f208b9ee98a/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.455709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6333b27309b4ad0b9429a8117996f9c, user_id=92c5b7cb92954678a5e66f208b9ee98a, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.462234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.485717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 978/1966] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:22 2022] PUT /identity/v3/projects/c6333b27309b4ad0b9429a8117996f9c/users/92c5b7cb92954678a5e66f208b9ee98a/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:22.503163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.503681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.504322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.565028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] MFA Rules not processed for user `92c5b7cb92954678a5e66f208b9ee98a`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:22.566646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:22.611514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:22.615926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:22.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 989/1967] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:22 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:22.757856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:22.759312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:22.791679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb93c14-9421-4729-b974-f50a6e952ced admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:22.800766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.802023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.802548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.803073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] PATH_INFO: `/v3/projects/1ddd62c6275e46c494a288e0be07f230` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.805386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ddd62c6275e46c494a288e0be07f230)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.809222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.817901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:22.848672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4e1ec7-0598-4ba6-94b3-6b18514697dd None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:22.855048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:22.856230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:22.856730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:22.857086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:22.876824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:22.879620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:22.933322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 990/1968] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:22 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:11:22.940156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ddd62c6275e46c494a288e0be07f230', 'request_id': 'req-99197a84-7bf5-43b7-8d4b-f91f04922a45'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:22.952126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ddd62c6275e46c494a288e0be07f230', 'request_id': 'req-99197a84-7bf5-43b7-8d4b-f91f04922a45'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:22.975864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ddd62c6275e46c494a288e0be07f230', 'request_id': 'req-99197a84-7bf5-43b7-8d4b-f91f04922a45'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.028630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] The token cache is being invalidate because project 1ddd62c6275e46c494a288e0be07f230 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:23.029407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.032488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.037078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 979/1969] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:22 2022] DELETE /identity/v3/projects/1ddd62c6275e46c494a288e0be07f230 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:23.042340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:23.058608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187021b6-743a-411b-b010-10bda1a2f73b service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:23.260381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:23.262842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:23.263580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:23.264297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] PATH_INFO: `/v3/users/343e70635e7749caadfe6b876283ddc3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:23.267007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=343e70635e7749caadfe6b876283ddc3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:23.272572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:23.376014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '343e70635e7749caadfe6b876283ddc3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.384971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '343e70635e7749caadfe6b876283ddc3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.385957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '343e70635e7749caadfe6b876283ddc3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.409015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '343e70635e7749caadfe6b876283ddc3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.443240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '343e70635e7749caadfe6b876283ddc3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:23.497796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 991/1970] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:23 2022] DELETE /identity/v3/users/343e70635e7749caadfe6b876283ddc3 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:24.604900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:24.608276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:24.642124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99197a84-7bf5-43b7-8d4b-f91f04922a45 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:24.729860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:24.732609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:24.734627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:24.852054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:24.853682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:24.854408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:24.855507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] PATH_INFO: `/v3/users/b308b1d021444ab1bacda522c3e277ef` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:24.860039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b308b1d021444ab1bacda522c3e277ef)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:24.864689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:24.907728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d403e077-0fd1-4a11-b4d3-fb1ddc1cf5e6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:24.916289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:24.917309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:24.917774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:24.918242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] PATH_INFO: `/v3/projects/1a1531dd993449e2b625964521acf7c0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:24.920801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a1531dd993449e2b625964521acf7c0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:24.924614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:24.929242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:25.001921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b308b1d021444ab1bacda522c3e277ef'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.021111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b308b1d021444ab1bacda522c3e277ef'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.045521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b308b1d021444ab1bacda522c3e277ef'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.047269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b308b1d021444ab1bacda522c3e277ef'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.049011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eae1746-bfb3-4780-9696-f2a6f4c725d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b308b1d021444ab1bacda522c3e277ef'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.056718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a1531dd993449e2b625964521acf7c0', 'request_id': 'req-1476210b-5c8a-4b5c-b40f-f702e942cacd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.071627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a1531dd993449e2b625964521acf7c0', 'request_id': 'req-1476210b-5c8a-4b5c-b40f-f702e942cacd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.088643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a1531dd993449e2b625964521acf7c0', 'request_id': 'req-1476210b-5c8a-4b5c-b40f-f702e942cacd'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.102802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 980/1971] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:24 2022] DELETE /identity/v3/users/b308b1d021444ab1bacda522c3e277ef => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:25.113349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:25.114013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:25.114834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:25.158301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] The token cache is being invalidate because project 1a1531dd993449e2b625964521acf7c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:25.160381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.165309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1476210b-5c8a-4b5c-b40f-f702e942cacd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:25.171556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 992/1972] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:24 2022] DELETE /identity/v3/projects/1a1531dd993449e2b625964521acf7c0 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:25.181970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d01b58-2a7b-412e-89d1-ff3936409b7a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:25.182472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d01b58-2a7b-412e-89d1-ff3936409b7a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:25.184003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d01b58-2a7b-412e-89d1-ff3936409b7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:25.202027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:25.210175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:25.277920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a2d01b58-2a7b-412e-89d1-ff3936409b7a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:25.280148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a2d01b58-2a7b-412e-89d1-ff3936409b7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:25.435504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 981/1973] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:25.451910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:25.457707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:25.496180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 993/1974] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:25.512368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:25.513352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:25.514304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:25.558549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:25.561209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:25.576455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 994/1975] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:25.588077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:25.602765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe67dd62-53d9-43d9-aa17-f150ab8510b4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:25.611715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:25.612981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:25.613572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:25.614012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:25.614891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:25.615767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:25.619441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:25.632178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e003c39-0f2d-4a9b-85dc-f9aa0d4e9b06 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:25.644568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:25.646209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:25.646439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 995/1976] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:25.652197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:25.652639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:25.654048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:25.664828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:25.676299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:25.679118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2ac013-a1b1-4d82-868a-105a9bb6a6e6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:25.682981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:25.688828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:25.690012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:25.691284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:25.692034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:25.693781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:25.698161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:25.708522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 982/1977] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:11:25.818932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 996/1978] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:25 2022] POST /identity/v3/projects => generated 375 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:11:25.824619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:25.851611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2360d798-eb38-4b06-9f62-ca4ad0d381ef service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:26.038798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:26.040149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.040701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.041178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.042413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:26.049669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:26.093959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 983/1979] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:25 2022] POST /identity/v3/users => generated 288 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:26.101067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.128723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46719c48-aeda-4eef-bab1-4460f91c9f83 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:26.141186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:26.142102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.142577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.144866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:26.148117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:26.171961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 997/1980] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:26 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:26.178066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.194800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b44658-6d01-4a3c-a0b1-1a2d659fbb96 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:26.200396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b4764e-2a5d-496e-9c8a-26d997965141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:26.201097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b4764e-2a5d-496e-9c8a-26d997965141 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.201453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b4764e-2a5d-496e-9c8a-26d997965141 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.201757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b4764e-2a5d-496e-9c8a-26d997965141 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.202412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b4764e-2a5d-496e-9c8a-26d997965141 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:26.204158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b4764e-2a5d-496e-9c8a-26d997965141 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:26.230119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 984/1981] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:26 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:11:26.233738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.248956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5251bb0-eb96-4b6c-b9de-7ba2b6dbf422 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:26.256673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:26.257711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.258002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.258283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] PATH_INFO: `/v3/projects/45266fa3efb047909225e48c3c121d99/users/9a5f29df9b5742cea2564048b9124ce4/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.295586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] RBAC: Authorizing `identity:create_grant(project_id=45266fa3efb047909225e48c3c121d99, user_id=9a5f29df9b5742cea2564048b9124ce4, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:26.298831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:26.330068 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 998/1982] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:26 2022] PUT /identity/v3/projects/45266fa3efb047909225e48c3c121d99/users/9a5f29df9b5742cea2564048b9124ce4/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:26.350375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.351069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.351763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.437907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] MFA Rules not processed for user `9a5f29df9b5742cea2564048b9124ce4`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:26.440647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:26.538754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.549433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:26.605470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:26.691670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 985/1983] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:26.694788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.766817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e297a99-4be1-4182-9173-b02250b5d740 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:26.778375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:26.780860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.781278 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.781740 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] PATH_INFO: `/v3/projects/49321854a38d457780ed1bac7afce5a5` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.792326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] RBAC: Authorizing `identity:delete_project(project_id=49321854a38d457780ed1bac7afce5a5)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:26.792326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:26.825356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aed54c-e8e4-4e00-ba35-52bfb2377e54 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:26.833478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:26.834737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:26.835214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:26.835688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:26.836958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:26.840771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:26.853455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:26.879797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49321854a38d457780ed1bac7afce5a5', 'request_id': 'req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:26.899650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49321854a38d457780ed1bac7afce5a5', 'request_id': 'req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:26.913326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 986/1984] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:26 2022] POST /identity/v3/projects => generated 375 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:26.915035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.922924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49321854a38d457780ed1bac7afce5a5', 'request_id': 'req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:26.949303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] The token cache is being invalidate because project 49321854a38d457780ed1bac7afce5a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:26.949678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:26.950809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:26.953291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 999/1985] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:26 2022] DELETE /identity/v3/projects/49321854a38d457780ed1bac7afce5a5 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:26.961556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:26.964010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:27.038348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab37988-2fa9-49eb-9fab-8ca6b0d14d09 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:27.051385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:27.052504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.053139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.053599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.054706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:27.059071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:27.069587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e083adf-6b91-40e0-9c3f-63f9c8e7107d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:27.076872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:27.077914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.078191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.078583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] PATH_INFO: `/v3/users/92c5b7cb92954678a5e66f208b9ee98a` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.080518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] RBAC: Authorizing `identity:delete_user(user_id=92c5b7cb92954678a5e66f208b9ee98a)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:27.083337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:27.106888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 987/1986] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:26 2022] POST /identity/v3/users => generated 294 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:27.108906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:27.112325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:27.137251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d10d96-0863-4a6b-bf08-4f0c58d0e02e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:27.146410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:27.147365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.147827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.148267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.149220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:27.153297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:27.182082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 988/1987] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:27 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:11:27.184210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:27.185011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:27.188645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92c5b7cb92954678a5e66f208b9ee98a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:27.190168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92c5b7cb92954678a5e66f208b9ee98a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:27.191697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92c5b7cb92954678a5e66f208b9ee98a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:27.208021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92c5b7cb92954678a5e66f208b9ee98a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:27.210093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a655dba2-002c-4fdd-aebd-4a4bc022bd82 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:27.220168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:27.224166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.224911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.225436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] PATH_INFO: `/v3/projects/fc3e73b9f5c44fb79b41dc2b213b639f/users/07b4cf1aa4b446d4b3ae2feeaa677314/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.228815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92c5b7cb92954678a5e66f208b9ee98a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:27.268318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1000/1988] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:26 2022] DELETE /identity/v3/users/92c5b7cb92954678a5e66f208b9ee98a => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:27.272114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc3e73b9f5c44fb79b41dc2b213b639f, user_id=07b4cf1aa4b446d4b3ae2feeaa677314, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:27.276435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:27.300314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 989/1989] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:27 2022] PUT /identity/v3/projects/fc3e73b9f5c44fb79b41dc2b213b639f/users/07b4cf1aa4b446d4b3ae2feeaa677314/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:27.306370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:27.309751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:27.479292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff626c-d584-47b9-9718-06df4187b651 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:27.585946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a16ce0-6c5d-4be5-9d2b-17ccdacdeeba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:27.603037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a16ce0-6c5d-4be5-9d2b-17ccdacdeeba admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.603037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a16ce0-6c5d-4be5-9d2b-17ccdacdeeba admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.604730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a16ce0-6c5d-4be5-9d2b-17ccdacdeeba admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.606780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a16ce0-6c5d-4be5-9d2b-17ccdacdeeba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:27.609489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a16ce0-6c5d-4be5-9d2b-17ccdacdeeba admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:27.636587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1001/1990] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:27 2022] GET /identity/v3/roles => generated 1420 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:27.642268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:27.657126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e176c146-d353-4add-a5f3-8b464e4d3732 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:27.664720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:27.665643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.665952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.666228 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] PATH_INFO: `/v3/projects/fc3e73b9f5c44fb79b41dc2b213b639f/users/07b4cf1aa4b446d4b3ae2feeaa677314/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.669252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc3e73b9f5c44fb79b41dc2b213b639f, user_id=07b4cf1aa4b446d4b3ae2feeaa677314, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:27.672922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:27.692276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 990/1991] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:27 2022] PUT /identity/v3/projects/fc3e73b9f5c44fb79b41dc2b213b639f/users/07b4cf1aa4b446d4b3ae2feeaa677314/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:27.714535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:27.714953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:27.715437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:27.772688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] MFA Rules not processed for user `07b4cf1aa4b446d4b3ae2feeaa677314`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:27.774880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:27.890809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:27.894930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:27.895547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:27.959966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1002/1992] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:27 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 247 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:27.969544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:27.972062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:28.054424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c944995b-b6c2-4344-bb5a-851562a14425 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:28.060393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:28.060984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.061270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.061549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.067744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:28.108333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408f2ef-1440-4c5b-9a8e-95b474749f69 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:28.117571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:28.119371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.119840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.120337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] PATH_INFO: `/v3/projects/c6333b27309b4ad0b9429a8117996f9c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.122851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6333b27309b4ad0b9429a8117996f9c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:28.126526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:28.207660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:INSERT=6,keystone:DELETE=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:28.222557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:28.222557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8bace3-004f-4509-92e5-fde8a4476936 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:28.246294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1003/1993] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:28.247072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6333b27309b4ad0b9429a8117996f9c', 'request_id': 'req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:28.259575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6333b27309b4ad0b9429a8117996f9c', 'request_id': 'req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:28.283723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6333b27309b4ad0b9429a8117996f9c', 'request_id': 'req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:28.323921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] The token cache is being invalidate because project c6333b27309b4ad0b9429a8117996f9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:28.324664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:28.326501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ba9c6429-bd1e-4b0d-99b6-246c3dbed486 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:28.335107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 991/1994] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:27 2022] DELETE /identity/v3/projects/c6333b27309b4ad0b9429a8117996f9c => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:28.337136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.337433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.337716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.368694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:28.370086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:28.486896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1004/1995] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:28.495448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.495842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.496190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.545535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:28.547635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:28.565656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 992/1996] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:28.573675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:28.594560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1164619b-5a4f-42e4-b0c8-d8c72e59ec84 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:28.610646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:28.611937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.613199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.613762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.616411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:28.617083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:28.621856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:28.646188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1005/1997] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:28.653038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:28.669168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cabc6b-05cf-4a00-9873-6135cd5db9a9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:28.677303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:28.678186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.678738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.679071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.679865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:28.683141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:28.751871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 993/1998] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:28 2022] POST /identity/v3/projects => generated 383 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:28.758148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:28.779844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd3fce-a1e5-4b78-9d75-38c2bf804ee1 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:28.856454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:28.904592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:28.905661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.906043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.906396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.907198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:28.911152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:28.947730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1006/1999] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:28 2022] POST /identity/v3/users => generated 292 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:28.955068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:28.985988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb0df99-587b-416c-9066-552cbc8b7107 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:28.992340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:28.993733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:28.994185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:28.994469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:28.995160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:28.997816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:29.020289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 994/2000] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:29.022704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:29.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7910b-9ef4-4d6a-b6e6-ebf693361b75 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:29.046550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97f9d53-e089-40cd-b628-d0fb1896ee9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:29.047224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97f9d53-e089-40cd-b628-d0fb1896ee9a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:29.047544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97f9d53-e089-40cd-b628-d0fb1896ee9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:29.047859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97f9d53-e089-40cd-b628-d0fb1896ee9a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:29.048674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97f9d53-e089-40cd-b628-d0fb1896ee9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:29.050489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97f9d53-e089-40cd-b628-d0fb1896ee9a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:29.062815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1007/2001] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:29 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:11:29.068395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:29.079874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ada45e3-28b5-4cf6-9308-ba9a3542a59b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:29.085886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:29.086682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:29.087114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:29.087493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] PATH_INFO: `/v3/projects/f5a3161b76cc434bb227e03a4ba8ffb8/users/197de453790f4c37970fc61234707ad0/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:29.107253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5a3161b76cc434bb227e03a4ba8ffb8, user_id=197de453790f4c37970fc61234707ad0, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:29.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:29.134089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 995/2002] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:29 2022] PUT /identity/v3/projects/f5a3161b76cc434bb227e03a4ba8ffb8/users/197de453790f4c37970fc61234707ad0/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:29.152797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:29.153346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:29.153736 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:29.193206 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] MFA Rules not processed for user `197de453790f4c37970fc61234707ad0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:29.194772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:29.296143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1008/2003] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:29.317175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:29.319804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:29.427447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aed1d47-a690-4567-bb1d-3ee44b6340e4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:29.435743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defa362f-69ec-4c76-8f75-a02aa115b63a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:29.436918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa362f-69ec-4c76-8f75-a02aa115b63a service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:29.437414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa362f-69ec-4c76-8f75-a02aa115b63a service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:29.437870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa362f-69ec-4c76-8f75-a02aa115b63a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:29.457435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defa362f-69ec-4c76-8f75-a02aa115b63a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:29.460479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defa362f-69ec-4c76-8f75-a02aa115b63a service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:29.484555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 996/2004] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:31.610692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:31.617156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:31.895632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda3be77-464c-4898-86f9-cbb75a7290d2 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:31.943146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:31.944265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:31.945026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:31.945602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] PATH_INFO: `/v3/users/9a5f29df9b5742cea2564048b9124ce4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:31.953288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a5f29df9b5742cea2564048b9124ce4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:31.956768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:32.148989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:32.157276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a5f29df9b5742cea2564048b9124ce4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.158777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a5f29df9b5742cea2564048b9124ce4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.161184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a5f29df9b5742cea2564048b9124ce4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.183390 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a5f29df9b5742cea2564048b9124ce4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.211823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a5f29df9b5742cea2564048b9124ce4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.259569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1009/2005] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:31 2022] DELETE /identity/v3/users/9a5f29df9b5742cea2564048b9124ce4 => generated 0 bytes in 650 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:32.267144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:32.268068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:32.268068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:32.319688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:32.322012 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:32.449082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:32.468848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 997/2006] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:32.485682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:32.488666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:32.592567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:32.597546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:32.611003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b11ce20-8944-461a-946e-c51281758d75 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:32.611318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d3d98-fc9b-41b2-a836-d559eca4a279 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:32.618193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b0d59-1592-40e5-bd85-28dbae461d84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:32.618985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b0d59-1592-40e5-bd85-28dbae461d84 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:32.619389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b0d59-1592-40e5-bd85-28dbae461d84 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:32.619896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b0d59-1592-40e5-bd85-28dbae461d84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:32.623257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:32.625764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:32.626356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:32.626839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] PATH_INFO: `/v3/users/322c9ac0ebdd413faeccf0bf54083e29` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:32.628736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=322c9ac0ebdd413faeccf0bf54083e29)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:32.633533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7b0d59-1592-40e5-bd85-28dbae461d84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:32.634118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:32.635164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7b0d59-1592-40e5-bd85-28dbae461d84 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:32.652936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1010/2007] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:32.743772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '322c9ac0ebdd413faeccf0bf54083e29'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.757596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '322c9ac0ebdd413faeccf0bf54083e29'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.774574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '322c9ac0ebdd413faeccf0bf54083e29'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.775725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '322c9ac0ebdd413faeccf0bf54083e29'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.776775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '322c9ac0ebdd413faeccf0bf54083e29'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:32.816380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 998/2008] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:32 2022] DELETE /identity/v3/users/322c9ac0ebdd413faeccf0bf54083e29 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:32.819888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:32.820746 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:32.821231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:32.879980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:32.879980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:33.015408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1011/2009] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:33.077505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:33.081087 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:33.125139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:33.127609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:33.137535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf73e748-2ce6-4232-9ff7-7c95360a7b65 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:33.150813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:33.152326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:33.152979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:33.153673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] PATH_INFO: `/v3/projects/45266fa3efb047909225e48c3c121d99` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:33.157347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45266fa3efb047909225e48c3c121d99)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:33.162158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:33.262370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45266fa3efb047909225e48c3c121d99', 'request_id': 'req-935082ba-d039-4d26-8526-53bb350ae0e9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.273461 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d892f5-fc14-4f23-8479-d62a3867cae7 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:33.274406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45266fa3efb047909225e48c3c121d99', 'request_id': 'req-935082ba-d039-4d26-8526-53bb350ae0e9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.297837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45266fa3efb047909225e48c3c121d99', 'request_id': 'req-935082ba-d039-4d26-8526-53bb350ae0e9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.324703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:33.326498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] The token cache is being invalidate because project 45266fa3efb047909225e48c3c121d99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:33.326838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.327766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.334439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 999/2010] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:33 2022] DELETE /identity/v3/projects/45266fa3efb047909225e48c3c121d99 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:33.335707 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:33.338540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:33.405704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:33.406899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:33.407364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:33.407913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:33.412756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:33.427473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:33.466782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935082ba-d039-4d26-8526-53bb350ae0e9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:33.473907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:33.474676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:33.474988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:33.475317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] PATH_INFO: `/v3/users/07b4cf1aa4b446d4b3ae2feeaa677314` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:33.476950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] RBAC: Authorizing `identity:delete_user(user_id=07b4cf1aa4b446d4b3ae2feeaa677314)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:33.481606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:33.586840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07b4cf1aa4b446d4b3ae2feeaa677314'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.616963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07b4cf1aa4b446d4b3ae2feeaa677314'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.639817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07b4cf1aa4b446d4b3ae2feeaa677314'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.642749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07b4cf1aa4b446d4b3ae2feeaa677314'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.645320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07b4cf1aa4b446d4b3ae2feeaa677314'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:33.669627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:33.672691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6a877b-eca9-4494-88b5-07413437606b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:33.698140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1000/2011] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:33 2022] DELETE /identity/v3/users/07b4cf1aa4b446d4b3ae2feeaa677314 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:33.703335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1012/2012] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 580 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:11:34.400840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:34.402553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:34.579395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525bfe4-4f10-489a-ab66-3c2bdaef197e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:34.598512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:34.599586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:34.600386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:34.601061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] PATH_INFO: `/v3/projects/fc3e73b9f5c44fb79b41dc2b213b639f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:34.603275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc3e73b9f5c44fb79b41dc2b213b639f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:34.605688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:34.687270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc3e73b9f5c44fb79b41dc2b213b639f', 'request_id': 'req-28b2fdfd-3aaa-4916-b734-dfae91b895a9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:34.700587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc3e73b9f5c44fb79b41dc2b213b639f', 'request_id': 'req-28b2fdfd-3aaa-4916-b734-dfae91b895a9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:34.717032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc3e73b9f5c44fb79b41dc2b213b639f', 'request_id': 'req-28b2fdfd-3aaa-4916-b734-dfae91b895a9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:34.742178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] The token cache is being invalidate because project fc3e73b9f5c44fb79b41dc2b213b639f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:34.742672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:34.743761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:34.746484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1001/2013] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:34 2022] DELETE /identity/v3/projects/fc3e73b9f5c44fb79b41dc2b213b639f => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:34.754368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec38c5a-fc78-4ecd-ab49-feb15877ca8c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:34.755164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec38c5a-fc78-4ecd-ab49-feb15877ca8c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:34.755164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec38c5a-fc78-4ecd-ab49-feb15877ca8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:34.780135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-eec38c5a-fc78-4ecd-ab49-feb15877ca8c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:34.781549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-eec38c5a-fc78-4ecd-ab49-feb15877ca8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:34.915523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:34.920153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:34.948435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:34.982061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1013/2014] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:34.985159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:34.990249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:34.992411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.063954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:35.073044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:35.097219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1014/2015] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:35.099747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:35.131428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960f3abc-5f8c-4a40-b8ad-e1f6195cb69f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:35.145662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:35.146802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:35.147285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:35.147827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.149200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:35.149734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:35.155966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:35.201379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b2fdfd-3aaa-4916-b734-dfae91b895a9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:35.202094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1015/2016] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:35.203424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:35.210564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:35.211970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:35.212584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:35.213162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] PATH_INFO: `/v3/projects/64218ac0bd314ac2af6f50b34fec5abd` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.215792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64218ac0bd314ac2af6f50b34fec5abd)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:35.223137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b476cf-40d4-47d8-a9e4-70b67f4ac688 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:35.232305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:35.246004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:35.246887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:35.247182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:35.247466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.248179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:35.252535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:35.333476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1016/2017] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:35 2022] POST /identity/v3/projects => generated 375 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:35.342047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:35.384437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64218ac0bd314ac2af6f50b34fec5abd', 'request_id': 'req-8f04aba1-384c-4cd7-b017-4114587b75b5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.406524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64218ac0bd314ac2af6f50b34fec5abd', 'request_id': 'req-8f04aba1-384c-4cd7-b017-4114587b75b5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.411559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:35.428853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64218ac0bd314ac2af6f50b34fec5abd', 'request_id': 'req-8f04aba1-384c-4cd7-b017-4114587b75b5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.460351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] The token cache is being invalidate because project 64218ac0bd314ac2af6f50b34fec5abd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:35.460850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.462828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.470018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1002/2018] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:34 2022] DELETE /identity/v3/projects/64218ac0bd314ac2af6f50b34fec5abd => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:35.474286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:35.477006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:35.489044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f056ea7-2247-4c0b-9613-b5b2e8434fad admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:35.614464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f04aba1-384c-4cd7-b017-4114587b75b5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:35.629478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:35.631071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:35.631838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:35.632352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] PATH_INFO: `/v3/users/b382e5f6f10441949fe0243c9a01ee08` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.634269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] RBAC: Authorizing `identity:delete_user(user_id=b382e5f6f10441949fe0243c9a01ee08)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:35.635477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:35.637258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:35.638248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:35.638619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:35.638918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.640036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:35.643863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:35.697400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1017/2019] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:35 2022] POST /identity/v3/users => generated 288 bytes in 357 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:35.705213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:35.709009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:35.718051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:35.756398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b382e5f6f10441949fe0243c9a01ee08'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.759664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a294fa-fb29-4e24-8b2f-4475165475b1 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:35.772965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:35.774469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b382e5f6f10441949fe0243c9a01ee08'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.775987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:35.777029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:35.778133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:35.786960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:35.790915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:35.805576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b382e5f6f10441949fe0243c9a01ee08'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.807355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b382e5f6f10441949fe0243c9a01ee08'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.808546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b382e5f6f10441949fe0243c9a01ee08'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:35.817458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1018/2020] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:35 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:35.823251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:35.827987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:35.855017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1003/2021] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:35 2022] DELETE /identity/v3/users/b382e5f6f10441949fe0243c9a01ee08 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:35.866665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6705565-c4ae-4aa8-a32f-6a74cf3db822 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:36.032528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5495d430-b5eb-40d1-b747-32bfe30a81a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:36.034486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5495d430-b5eb-40d1-b747-32bfe30a81a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:36.034486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5495d430-b5eb-40d1-b747-32bfe30a81a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:36.034841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5495d430-b5eb-40d1-b747-32bfe30a81a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:36.035773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5495d430-b5eb-40d1-b747-32bfe30a81a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:36.040066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5495d430-b5eb-40d1-b747-32bfe30a81a2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:36.080193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1019/2022] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:35 2022] GET /identity/v3/roles => generated 1420 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:11:36.082149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:36.118348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8ef227-9779-4ca0-a933-62d4e889604e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:36.146712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:36.147464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:36.147751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:36.148029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] PATH_INFO: `/v3/projects/8034d44cbe8b49c1bd2f32aae4e4807d/users/4841e6983110466c8df90c3e269de4ac/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:36.189307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8034d44cbe8b49c1bd2f32aae4e4807d, user_id=4841e6983110466c8df90c3e269de4ac, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:36.196222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:36.240667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1004/2023] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:36 2022] PUT /identity/v3/projects/8034d44cbe8b49c1bd2f32aae4e4807d/users/4841e6983110466c8df90c3e269de4ac/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:36.264966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:36.266046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:36.266792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:36.388281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] MFA Rules not processed for user `4841e6983110466c8df90c3e269de4ac`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:36.393177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:36.580168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:36.718410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1020/2024] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 456 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:36.723958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:36.998750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.001861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:37.040006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71be951-8221-4930-893a-8361c1971fc9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.047916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9336588e-3169-40a9-b5b9-77f668344776 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.060704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.061924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.062428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.062930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.064104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.066216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.068281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.069082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.070046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] PATH_INFO: `/v3/projects/e5bd676288ef46ffb7840de3a3695734` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.070795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.073325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5bd676288ef46ffb7840de3a3695734)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.075997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1005/2025] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:36 2022] POST /identity/v3/projects => generated 375 bytes in 462 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:37.195309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.210701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5bd676288ef46ffb7840de3a3695734', 'request_id': 'req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.236601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5bd676288ef46ffb7840de3a3695734', 'request_id': 'req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.262844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5bd676288ef46ffb7840de3a3695734', 'request_id': 'req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.293757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] The token cache is being invalidate because project e5bd676288ef46ffb7840de3a3695734 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:37.294768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.296399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.303980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1021/2026] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:36 2022] DELETE /identity/v3/projects/e5bd676288ef46ffb7840de3a3695734 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:37.303980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.304877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:37.370497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcebbbf-fb09-4b01-b389-a1db3a2327e1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.376974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e1ba40-66f8-484b-8f99-40b994d3d29c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.377827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e1ba40-66f8-484b-8f99-40b994d3d29c admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.378370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e1ba40-66f8-484b-8f99-40b994d3d29c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.378828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e1ba40-66f8-484b-8f99-40b994d3d29c admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.379904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e1ba40-66f8-484b-8f99-40b994d3d29c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.382637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e1ba40-66f8-484b-8f99-40b994d3d29c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.426774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1006/2027] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:37 2022] POST /identity/v3/users => generated 288 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:37.426774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.426774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.426774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.428886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e794c7f0-73fd-41d9-a4c8-a40c17a2bb0b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.437161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.437872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.438185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.438590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] PATH_INFO: `/v3/users/caafb1ccfd704e3ca65b6c4db04b05ef` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.440502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=caafb1ccfd704e3ca65b6c4db04b05ef)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.443088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.445804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=5,keystone:INSERT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:37.466541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:37.468167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:37.479032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1007/2028] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:37.480172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.491300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:37.514775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642a0baf-6073-4a54-8ac4-590da3538d35 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.521860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fdd5e8-f4cf-468a-aef0-5ade73071c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.522529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdd5e8-f4cf-468a-aef0-5ade73071c7a admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.522807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdd5e8-f4cf-468a-aef0-5ade73071c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.523085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdd5e8-f4cf-468a-aef0-5ade73071c7a admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.523895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fdd5e8-f4cf-468a-aef0-5ade73071c7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.525805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fdd5e8-f4cf-468a-aef0-5ade73071c7a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.546553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1008/2029] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:37.546553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.546553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.546553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.560226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'caafb1ccfd704e3ca65b6c4db04b05ef'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.561496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'caafb1ccfd704e3ca65b6c4db04b05ef'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.562761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'caafb1ccfd704e3ca65b6c4db04b05ef'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.578311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'caafb1ccfd704e3ca65b6c4db04b05ef'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.584648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:37.586905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:37.590514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'caafb1ccfd704e3ca65b6c4db04b05ef'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:37.594367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:37.610441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1009/2030] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:37.614224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.617018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:37.628118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1022/2031] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:37 2022] DELETE /identity/v3/users/caafb1ccfd704e3ca65b6c4db04b05ef => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:37.629831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.642204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5e8bc0-21ab-4b67-bad0-0b0088f34833 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.772068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.773877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.773877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.774637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.776190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.776921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285c67e5-d693-46fc-8fe8-c9d845888fd5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.779695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.784742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.785877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.786470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.786972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.788205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:37.789070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.791956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.801197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1010/2032] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:37 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:37.807721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.820795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1023/2033] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:37.826391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.826692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d1982-f3db-4c8f-9b2d-146a4dda3cda admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.836964 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca77ce1-0e9b-4cef-8769-1d45579b92e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.838190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca77ce1-0e9b-4cef-8769-1d45579b92e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.838750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca77ce1-0e9b-4cef-8769-1d45579b92e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.839264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca77ce1-0e9b-4cef-8769-1d45579b92e9 admin admin] PATH_INFO: `/v3/projects/537b6516b5274c98b1b6255df9c1c3ae/users/16dcbdd29ad0443a9a4fc731372804df/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.849294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd69c1fe-b72a-4c9f-b899-2fc8720d8117 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:37.860944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:37.861934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.863045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.863467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.864660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.866558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca77ce1-0e9b-4cef-8769-1d45579b92e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=537b6516b5274c98b1b6255df9c1c3ae, user_id=16dcbdd29ad0443a9a4fc731372804df, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:37.868581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.871829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca77ce1-0e9b-4cef-8769-1d45579b92e9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:37.897109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1011/2034] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:37 2022] PUT /identity/v3/projects/537b6516b5274c98b1b6255df9c1c3ae/users/16dcbdd29ad0443a9a4fc731372804df/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:37.913502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:37.913986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:37.914441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:37.930219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1024/2035] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:37 2022] POST /identity/v3/projects => generated 367 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:11:37.936844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:37.961029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] MFA Rules not processed for user `16dcbdd29ad0443a9a4fc731372804df`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:37.963211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.063514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80698ee5-f280-44f9-8fec-31311f126ec7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.071074 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.071868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.072200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.072623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.073367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.075751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.091835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1012/2036] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:38.092773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.094169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.111455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463bebbf-6031-4620-a1a0-58f26d6f7f55 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.118866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.119561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.119999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.120123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] PATH_INFO: `/v3/projects/a105c95deb1041ba9ac721d7c82110e9` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.120975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1025/2037] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:37 2022] POST /identity/v3/users => generated 284 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:38.121565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a105c95deb1041ba9ac721d7c82110e9)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.122607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.124507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.137104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39681baa-fdac-4383-9793-942e150a52f8 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.147594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.149089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.149443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.149790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.150524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.153277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.182824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:38.198011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1026/2038] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/projects => generated 377 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:38.198011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.204667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a105c95deb1041ba9ac721d7c82110e9', 'request_id': 'req-1735d7ea-b774-4346-a48e-40654007a27e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:38.214100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a105c95deb1041ba9ac721d7c82110e9', 'request_id': 'req-1735d7ea-b774-4346-a48e-40654007a27e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:38.230907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a105c95deb1041ba9ac721d7c82110e9', 'request_id': 'req-1735d7ea-b774-4346-a48e-40654007a27e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:38.247650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] The token cache is being invalidate because project a105c95deb1041ba9ac721d7c82110e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:38.247936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:38.248758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:38.250457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1013/2039] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:38 2022] DELETE /identity/v3/projects/a105c95deb1041ba9ac721d7c82110e9 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:38.254584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.254584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.258249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=93,keystone:DELETE=2,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:38.333920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c030ad66-1d37-49ab-bf7e-29ba65a7295d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.340165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36710391-f445-4294-ad58-32194ac766a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.340958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36710391-f445-4294-ad58-32194ac766a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.341300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36710391-f445-4294-ad58-32194ac766a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.341644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36710391-f445-4294-ad58-32194ac766a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.342383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36710391-f445-4294-ad58-32194ac766a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.345041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36710391-f445-4294-ad58-32194ac766a9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.356398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735d7ea-b774-4346-a48e-40654007a27e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.359798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1027/2040] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:38 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:38.361488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.361874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.362264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.364021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.364868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.365298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.365739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.366582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.369133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.391477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:38.392759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.400291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1014/2041] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/users => generated 295 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:38.400291 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.400633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.406949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1028/2042] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:38.414220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.416148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f62a962-883b-4913-b9c2-57a2dd2c43e0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.422112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f4642b-1d16-49d9-a3b6-e5bfb31752f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.422854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4642b-1d16-49d9-a3b6-e5bfb31752f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.423218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4642b-1d16-49d9-a3b6-e5bfb31752f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.424079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4642b-1d16-49d9-a3b6-e5bfb31752f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.424402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f4642b-1d16-49d9-a3b6-e5bfb31752f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.427049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f4642b-1d16-49d9-a3b6-e5bfb31752f4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.432198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4774fdce-48e7-4d04-82ee-d90041301727 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.439390 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.440284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.440691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.441142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.442025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1015/2043] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:38 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:38.446098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.446215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.446215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.446364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.446442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.462886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1029/2044] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:38 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:38.464116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.474873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:38.476838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7954898-37fd-4c83-aa88-fc6b4d97479f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.477432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.482516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.483199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.483479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.483762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] PATH_INFO: `/v3/projects/479d523b7f064490b60cb6d0182c38ca/users/f229ba62346647a6ac2caa0ceff0c9f4/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.489525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1016/2045] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:38.491020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.505723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d1b1e5-3f62-4e4c-9a7e-db31afa66ea2 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.507637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] RBAC: Authorizing `identity:create_grant(project_id=479d523b7f064490b60cb6d0182c38ca, user_id=f229ba62346647a6ac2caa0ceff0c9f4, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.513696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.514659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c516636-ed2f-4f27-934f-e7eae4c16a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.515697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c516636-ed2f-4f27-934f-e7eae4c16a26 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c516636-ed2f-4f27-934f-e7eae4c16a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c516636-ed2f-4f27-934f-e7eae4c16a26 admin admin] PATH_INFO: `/v3/projects/aeab60c25f47457887113c16454e3b0d/users/eb04301ebf3842c5b0c08cbdc0e5159f/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.537989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1030/2046] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:38 2022] PUT /identity/v3/projects/479d523b7f064490b60cb6d0182c38ca/users/f229ba62346647a6ac2caa0ceff0c9f4/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:38.539647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.546617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c516636-ed2f-4f27-934f-e7eae4c16a26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aeab60c25f47457887113c16454e3b0d, user_id=eb04301ebf3842c5b0c08cbdc0e5159f, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.552457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c516636-ed2f-4f27-934f-e7eae4c16a26 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.583934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1017/2047] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:38 2022] PUT /identity/v3/projects/aeab60c25f47457887113c16454e3b0d/users/eb04301ebf3842c5b0c08cbdc0e5159f/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:38.587036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.589305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.589766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.637414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] MFA Rules not processed for user `f229ba62346647a6ac2caa0ceff0c9f4`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:38.639345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:38.642334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd30252-a55f-495f-ac3a-60bd0d79e29b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.741581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=5,keystone:SELECT=90,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:38.749681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1018/2048] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:38.751443 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.766848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.767912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.768969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.769469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.770738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:38.771302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.774440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.803276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1031/2049] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:38.804976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.839841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8828091-8b04-4e5e-8b99-ca297eae40f0 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.851653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.853155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.854032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.854511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.855620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.860294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.901118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a3c49-623e-4256-86a1-93857603dd85 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:38.908041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:38.911805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:38.912966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:38.913685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:38.914170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:38.915347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:38.918341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:38.943488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1032/2050] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/projects => generated 369 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:38.945507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.946965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1019/2051] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:38 2022] GET /identity/v3/roles => generated 1420 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:38.953748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:38.990990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b94f7f-6c3e-49da-9f63-7a7e2a1914fe admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.126210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66a1b4e-6f48-412e-a683-767e60447bab admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.132082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.134122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.134122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.134629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.136331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.139960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.141006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.141778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.142312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.142889 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.144418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.148353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.194078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1020/2052] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/users => generated 291 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:39.195766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.212336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1033/2053] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:38 2022] POST /identity/v3/projects => generated 393 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:39.214528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.222774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d5632-af55-4aa0-88a1-52b6ad4844cd admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.358832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e36844-3c0d-4003-bc28-a5649e232636 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.371595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.372873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.373357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.373809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] PATH_INFO: `/v3/projects/aeab60c25f47457887113c16454e3b0d/users/eb04301ebf3842c5b0c08cbdc0e5159f/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.377364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aeab60c25f47457887113c16454e3b0d, user_id=eb04301ebf3842c5b0c08cbdc0e5159f, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.377959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.378417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.378851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.384849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.385127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.386711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.404163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1021/2054] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:39 2022] PUT /identity/v3/projects/aeab60c25f47457887113c16454e3b0d/users/eb04301ebf3842c5b0c08cbdc0e5159f/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:39.405687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.406372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1034/2055] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:39.415086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.435986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043a8be5-924d-4ec7-af71-a77d39ef6b67 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.452043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:39.517755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44e45c1-9290-48b4-91b3-94cafadaa6b8 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.527433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a49cc33-34f5-427f-a6c0-74e267176138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.528072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49cc33-34f5-427f-a6c0-74e267176138 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.528547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49cc33-34f5-427f-a6c0-74e267176138 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.528827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a49cc33-34f5-427f-a6c0-74e267176138 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.529486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a49cc33-34f5-427f-a6c0-74e267176138 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.531775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a49cc33-34f5-427f-a6c0-74e267176138 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.561399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1022/2056] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:39 2022] POST /identity/v3/users => generated 297 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:39.563011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.563395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.563774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.578727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.579555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.579940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.581721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] PATH_INFO: `/v3/projects/9803d3b3c39743a1803d1169e957e2f5/users/4e5d4a1e711f4c2d88760f4f01a57905/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.600936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] MFA Rules not processed for user `eb04301ebf3842c5b0c08cbdc0e5159f`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:39.603577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:39.603818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9803d3b3c39743a1803d1169e957e2f5, user_id=4e5d4a1e711f4c2d88760f4f01a57905, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.607373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.624418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:39.634317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1035/2057] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:39 2022] PUT /identity/v3/projects/9803d3b3c39743a1803d1169e957e2f5/users/4e5d4a1e711f4c2d88760f4f01a57905/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:39.638320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.734130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1023/2058] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:39 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:39.735734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.756898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc1335-c665-4c2e-ad3b-4fd470b4f099 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.763376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2895bb-4f70-4782-b651-77695401be0c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.764350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.765357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.765862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.766299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.767406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.769808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.770895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.771599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.771865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.772142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.772862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.775171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.790391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1024/2059] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:39.790391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.790718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1036/2060] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:39.792218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:39.793114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.807414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e627f836-234d-4022-a1be-fe12f4af916a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.812657 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.813352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.813637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.813923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] PATH_INFO: `/v3/projects/9803d3b3c39743a1803d1169e957e2f5/users/4e5d4a1e711f4c2d88760f4f01a57905/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.816175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9803d3b3c39743a1803d1169e957e2f5, user_id=4e5d4a1e711f4c2d88760f4f01a57905, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.819402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.831545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1037/2061] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:39 2022] PUT /identity/v3/projects/9803d3b3c39743a1803d1169e957e2f5/users/4e5d4a1e711f4c2d88760f4f01a57905/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:39.832553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:39.918572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c852143-a748-4848-88ed-58646c22219b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.939627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccef9111-cf98-49cf-8e1c-85760f391f62 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:39.945229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b284e5-ee72-46c1-a655-88704760f160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:39.945958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b284e5-ee72-46c1-a655-88704760f160 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.946319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b284e5-ee72-46c1-a655-88704760f160 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.946662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b284e5-ee72-46c1-a655-88704760f160 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.947336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b284e5-ee72-46c1-a655-88704760f160 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:39.949023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b284e5-ee72-46c1-a655-88704760f160 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:39.961115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1038/2062] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:39.962661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:39.962954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:39.963268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:39.996528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:40.037850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:40.038927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.039419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.040077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.041373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] MFA Rules not processed for user `4e5d4a1e711f4c2d88760f4f01a57905`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:40.043139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.043519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.151602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:40.154550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:40.173754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1025/2063] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:40.175734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:40.195518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1039/2064] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:39 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:40.195518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:40.195518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.196640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d7247-fa97-4df7-9d6e-d9a3d3cd316a service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:40.200692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:40.201166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.201453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.201737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] PATH_INFO: `/v3/projects/9badaa18dcf64810b10dd7bf03d8e40d/users/ec931c9cc56a462991413926d34dfa46/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.213990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:40.224671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f09ddca-68ad-42b9-b0eb-12b0fd89e48e None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:40.238343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9badaa18dcf64810b10dd7bf03d8e40d, user_id=ec931c9cc56a462991413926d34dfa46, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:40.241301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:40.242368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.242853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.243328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] PATH_INFO: `/v3/users/197de453790f4c37970fc61234707ad0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.244096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:40.246196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=197de453790f4c37970fc61234707ad0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:40.251785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:40.270575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1026/2065] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:40 2022] PUT /identity/v3/projects/9badaa18dcf64810b10dd7bf03d8e40d/users/ec931c9cc56a462991413926d34dfa46/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:40.278683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:40.281660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.360118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '197de453790f4c37970fc61234707ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:40.361388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '197de453790f4c37970fc61234707ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:40.362978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '197de453790f4c37970fc61234707ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:40.375832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '197de453790f4c37970fc61234707ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:40.389486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-82b8192b-ed90-4844-8aa9-84750a2e01b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '197de453790f4c37970fc61234707ad0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:40.418628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7990d785-9de3-4326-a778-ccd8a4ac0bf0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:40.426523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:40.427491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.427979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.428494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.431569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.436372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1040/2066] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:40 2022] DELETE /identity/v3/users/197de453790f4c37970fc61234707ad0 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:40.438904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af51e28d-9f72-47f5-99d2-77fcd229e2ac None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.439473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af51e28d-9f72-47f5-99d2-77fcd229e2ac None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.439948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af51e28d-9f72-47f5-99d2-77fcd229e2ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.472685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-af51e28d-9f72-47f5-99d2-77fcd229e2ac None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:40.474047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-af51e28d-9f72-47f5-99d2-77fcd229e2ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.570868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1041/2067] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:40.574232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:40.576817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.577053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:40.577307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.577752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.602204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1027/2068] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:40.603293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:40.605785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.618058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:40.619536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.649157 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:40.656327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1042/2069] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:40.677634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:40.684096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:40.811911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a1d5e-a542-4d94-99a7-d5ba7e3736b3 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:40.936798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed24619d-f999-41e8-8d2e-7cf69732e388 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:40.949175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:40.951170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.952717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.954450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.967362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:40.968649 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:40.969117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:40.969528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:40.985192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:40.992096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:40.993008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:40.995103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:41.016084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1028/2070] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:41.017763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:41.020645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:41.032553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1043/2071] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:41.247111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632431e-c6bd-4227-af4c-6c77915872e4 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:41.255391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:41.256491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:41.257060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:41.257546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:41.262146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:41.435905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:41.439420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c7ae4-0462-4e6f-bef9-620c5445975a service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:41.459740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1029/2072] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:41 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:41.685876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:41.689019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:41.723090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee55b2c-54bc-4888-b26a-3ac2b4d80247 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:41.733663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:41.734793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:41.735298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:41.735793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] PATH_INFO: `/v3/projects/f5a3161b76cc434bb227e03a4ba8ffb8` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:41.738214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5a3161b76cc434bb227e03a4ba8ffb8)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:41.743619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:41.762079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:41.866372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5a3161b76cc434bb227e03a4ba8ffb8', 'request_id': 'req-de366242-b71d-4b62-8002-252c18a8d842'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:41.890561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5a3161b76cc434bb227e03a4ba8ffb8', 'request_id': 'req-de366242-b71d-4b62-8002-252c18a8d842'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:41.913307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5a3161b76cc434bb227e03a4ba8ffb8', 'request_id': 'req-de366242-b71d-4b62-8002-252c18a8d842'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:41.950230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] The token cache is being invalidate because project f5a3161b76cc434bb227e03a4ba8ffb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:41.950902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:41.953583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-de366242-b71d-4b62-8002-252c18a8d842 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:41.958826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1044/2073] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:41 2022] DELETE /identity/v3/projects/f5a3161b76cc434bb227e03a4ba8ffb8 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:41.967488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:41.968377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:41.968854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:42.055824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:42.067085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:42.261035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:42.407320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1030/2074] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:42.428064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:42.428606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:42.429013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:42.530670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:42.531588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:42.551713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:42.554360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:42.570953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1045/2075] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:42.574102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:42.590048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933b5c2-64ab-4542-8385-3d88f211253f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:42.597870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:42.598521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:42.598835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:42.599217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:42.600001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:42.600402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:42.604025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:42.635808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1046/2076] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:42.645868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:42.667920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6551d770-403b-483c-88db-448cbde27719 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:42.690987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:42.693939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:42.694316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:42.695138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:42.696017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:42.702827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:42.736632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fbbca9-7c3d-466a-b9b4-fcaec67cb819 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:42.752000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:42.757121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:42.757941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:42.758439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:42.761785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:42.783274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1047/2077] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:42 2022] POST /identity/v3/projects => generated 371 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:11:42.791359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:42.894628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:42.898566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:42.917501 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1031/2078] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:42.946676 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6357294-a3ec-46e0-849f-e12caf96d413 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:42.958286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:42.960589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:42.961626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:42.963425 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:42.964677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:42.968395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:43.024519 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:43.027385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1048/2079] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:42 2022] POST /identity/v3/users => generated 286 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:43.055914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99257b87-23dc-40ec-bee4-92693598bb87 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:43.064201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:43.065447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:43.065957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:43.066379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:43.067664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:43.074811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:43.104551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1032/2080] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:43.107091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:43.122831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83df47eb-bbbd-4bcc-88e7-bce0a2b66258 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:43.131427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a879e03-e41e-452d-909b-025d6da07204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:43.132351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a879e03-e41e-452d-909b-025d6da07204 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:43.132797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a879e03-e41e-452d-909b-025d6da07204 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:43.133166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a879e03-e41e-452d-909b-025d6da07204 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:43.134598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a879e03-e41e-452d-909b-025d6da07204 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:43.137078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a879e03-e41e-452d-909b-025d6da07204 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:43.152784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1049/2081] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:43 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:11:43.158701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:43.172900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd533e0-0721-43d2-bef3-3fc85e3549ac admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:43.182257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:43.184342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:43.184872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:43.185593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] PATH_INFO: `/v3/projects/513191b6f2a54b479687e09408c1989d/users/806634a2bab64f9da24dda8048c73c3a/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:43.220428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=513191b6f2a54b479687e09408c1989d, user_id=806634a2bab64f9da24dda8048c73c3a, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:43.225495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:43.257428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1033/2082] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:43 2022] PUT /identity/v3/projects/513191b6f2a54b479687e09408c1989d/users/806634a2bab64f9da24dda8048c73c3a/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:43.281916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa26e52-164c-4aa3-bacc-667c99824c65 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:43.281916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa26e52-164c-4aa3-bacc-667c99824c65 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:43.281916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa26e52-164c-4aa3-bacc-667c99824c65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:43.343904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-afa26e52-164c-4aa3-bacc-667c99824c65 None None] MFA Rules not processed for user `806634a2bab64f9da24dda8048c73c3a`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:43.347883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-afa26e52-164c-4aa3-bacc-667c99824c65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:43.431594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:43.559946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1050/2083] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:43 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:43.580832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:43.584329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:43.693528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:43.694381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:43.694577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:43.740897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] MFA Rules not processed for user `dfc9712739724c91bd9dd5fdaaebd642`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:43.748934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:43.777133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:43.885544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292beebe-fe11-43d7-bdda-211d38d76d30 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:43.894672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:43.895673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:43.899071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:43.900967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:43.935361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:43.940670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:43.975875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1034/2084] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:43 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:43.976612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1051/2085] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:43 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:44.658725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:44.662482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:44.833494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4b877b-0341-4565-95af-ac17424d7f91 service glance] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:44.840666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:44.842205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:44.842938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:44.843719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:44.848309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:45.059434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:45.061591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35247336-00bd-4cbb-8b2c-7298a2759c85 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:45.128593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1035/2086] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:44 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:11:47.662504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:47.664098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:47.802050 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c85e4d-7a1b-4f26-a4c0-a0b02fcbcee0 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:47.809002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:47.810085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:47.810874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:47.811786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] PATH_INFO: `/v3/users/f229ba62346647a6ac2caa0ceff0c9f4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:47.814094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=f229ba62346647a6ac2caa0ceff0c9f4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:47.818320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:47.950077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f229ba62346647a6ac2caa0ceff0c9f4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:47.951784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f229ba62346647a6ac2caa0ceff0c9f4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:47.952557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f229ba62346647a6ac2caa0ceff0c9f4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:47.969786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f229ba62346647a6ac2caa0ceff0c9f4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:47.984305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f229ba62346647a6ac2caa0ceff0c9f4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:47.995522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:48.021241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1052/2087] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:47 2022] DELETE /identity/v3/users/f229ba62346647a6ac2caa0ceff0c9f4 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:48.026142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:48.027074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:48.027074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:48.079139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:48.084653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:48.298571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1036/2088] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:48.329429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:48.332341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:48.488645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed32b282-0295-491d-b9a5-e375202abf7c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:48.496591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:48.497826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:48.498448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:48.499075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:48.521200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:48.525444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:48.566574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1053/2089] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:49.632181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:49.634765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:49.687289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d255f4-abfb-4077-82d7-be6576bca81a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:49.709472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:49.710567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:49.711038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:49.711604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] PATH_INFO: `/v3/projects/479d523b7f064490b60cb6d0182c38ca` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:49.743260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=479d523b7f064490b60cb6d0182c38ca)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:49.747481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:49.816595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:49.897472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '479d523b7f064490b60cb6d0182c38ca', 'request_id': 'req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:49.910783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '479d523b7f064490b60cb6d0182c38ca', 'request_id': 'req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:49.928963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '479d523b7f064490b60cb6d0182c38ca', 'request_id': 'req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:49.975815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] The token cache is being invalidate because project 479d523b7f064490b60cb6d0182c38ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:49.977489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:49.979148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:49.986044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1037/2090] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:49 2022] DELETE /identity/v3/projects/479d523b7f064490b60cb6d0182c38ca => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:49.996302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:50.029806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0c4f3-ab76-4e96-80d5-a17db1ca3578 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:50.208007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:50.210656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:50.211488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:50.212209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] PATH_INFO: `/v3/users/4e5d4a1e711f4c2d88760f4f01a57905` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:50.215901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e5d4a1e711f4c2d88760f4f01a57905)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:50.221558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:50.413346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e5d4a1e711f4c2d88760f4f01a57905'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:50.418299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e5d4a1e711f4c2d88760f4f01a57905'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:50.420895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e5d4a1e711f4c2d88760f4f01a57905'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:50.450850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e5d4a1e711f4c2d88760f4f01a57905'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:50.494549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1f5fc5f6-a2aa-473c-8e64-3f7c09c34e8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e5d4a1e711f4c2d88760f4f01a57905'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:50.583147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1054/2091] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:49 2022] DELETE /identity/v3/users/4e5d4a1e711f4c2d88760f4f01a57905 => generated 0 bytes in 590 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:51.146517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:51.149555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:51.202233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b50d1b-d1bc-49b3-a420-9298a8af99a8 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:51.419068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:51.420310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:51.420812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:51.421322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] PATH_INFO: `/v3/users/4841e6983110466c8df90c3e269de4ac` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:51.423638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=4841e6983110466c8df90c3e269de4ac)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:51.428727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:51.609422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4841e6983110466c8df90c3e269de4ac'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:51.626539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4841e6983110466c8df90c3e269de4ac'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:51.653368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4841e6983110466c8df90c3e269de4ac'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:51.655208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4841e6983110466c8df90c3e269de4ac'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:51.656701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4841e6983110466c8df90c3e269de4ac'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:51.735646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1038/2092] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:51 2022] DELETE /identity/v3/users/4841e6983110466c8df90c3e269de4ac => generated 0 bytes in 590 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:51.742457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:51.743305 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:51.743450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:51.787734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:51.789173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:51.797029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:51.973041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1055/2093] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:52.038463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:52.046452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:52.243889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:52.350718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f554a4ad-13fb-455d-8fba-bfeaf12519cd admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:52.372595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0543bffe-684f-47c4-9d4e-b4ea57bd3fae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:52.373620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0543bffe-684f-47c4-9d4e-b4ea57bd3fae service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:52.374095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0543bffe-684f-47c4-9d4e-b4ea57bd3fae service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:52.374570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0543bffe-684f-47c4-9d4e-b4ea57bd3fae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:52.391138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0543bffe-684f-47c4-9d4e-b4ea57bd3fae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:52.394266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0543bffe-684f-47c4-9d4e-b4ea57bd3fae service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:52.408880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1039/2094] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:52.646504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:52.649787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:52.801451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00068f5c-d45a-4665-90e1-68735caa6d51 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:52.835419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:52.842524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:52.842524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:52.842887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] PATH_INFO: `/v3/projects/9803d3b3c39743a1803d1169e957e2f5` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:52.854389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] RBAC: Authorizing `identity:delete_project(project_id=9803d3b3c39743a1803d1169e957e2f5)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:52.863469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:53.027120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9803d3b3c39743a1803d1169e957e2f5', 'request_id': 'req-f582e67e-954b-4de7-80b5-a9e0bf502adc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:53.063964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9803d3b3c39743a1803d1169e957e2f5', 'request_id': 'req-f582e67e-954b-4de7-80b5-a9e0bf502adc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:53.105660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9803d3b3c39743a1803d1169e957e2f5', 'request_id': 'req-f582e67e-954b-4de7-80b5-a9e0bf502adc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:53.155867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] The token cache is being invalidate because project 9803d3b3c39743a1803d1169e957e2f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:53.156384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:53.161521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f582e67e-954b-4de7-80b5-a9e0bf502adc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:53.164455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1056/2095] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:52 2022] DELETE /identity/v3/projects/9803d3b3c39743a1803d1169e957e2f5 => generated 0 bytes in 520 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:53.177473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:53.177473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:53.177473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:53.255259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:53.266423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:53.519663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1040/2096] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:53.535507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:53.535817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:53.536224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:53.622256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:53.631151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:53.664183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1057/2097] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:53.665742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:53.688131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dcb17e-03d4-44ad-89b2-201ce2ebaa39 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:53.713697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:53.714759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:53.715312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:53.716904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:53.724518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:53.725232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:53.783326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:53.854069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1041/2098] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:11:53.860912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:53.884702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97c2f1f-5d2a-4230-99ca-9a280cf4cc48 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:53.900167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3700a7-3e71-4c4e-9592-a84625c3506d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:53.903395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3700a7-3e71-4c4e-9592-a84625c3506d admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:53.905214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3700a7-3e71-4c4e-9592-a84625c3506d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:53.905668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3700a7-3e71-4c4e-9592-a84625c3506d admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:53.906851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3700a7-3e71-4c4e-9592-a84625c3506d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:53.919374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3700a7-3e71-4c4e-9592-a84625c3506d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:54.051002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1058/2099] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:53 2022] POST /identity/v3/projects => generated 373 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:54.057495 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:54.112728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd6d0e-2711-49f8-8263-b86a66df65d0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:54.258336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986f1513-3f1a-43c1-83cf-dae5cc9a213d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.258336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986f1513-3f1a-43c1-83cf-dae5cc9a213d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.258883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986f1513-3f1a-43c1-83cf-dae5cc9a213d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.315266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:54.317440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.318507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.319198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.320732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:54.325123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:54.403224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1042/2100] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:54 2022] POST /identity/v3/users => generated 287 bytes in 346 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:11:54.405123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:54.420802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-986f1513-3f1a-43c1-83cf-dae5cc9a213d None None] MFA Rules not processed for user `ec931c9cc56a462991413926d34dfa46`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:54.423244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-986f1513-3f1a-43c1-83cf-dae5cc9a213d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:54.455089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e26a8b5-d01e-457a-9adc-c55db0fb3660 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:54.473763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:54.475111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.475547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.476285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.477629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:54.481168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:54.566208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1043/2101] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:54 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:54.571949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:54.617614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fabb687-28c1-4a7e-bc59-77dd43898cbb admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:54.639936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:54.641751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:54.642883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.643197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.644053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.645001 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:54.661359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:54.696095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1059/2102] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 442 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:54.718755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1044/2103] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:54 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:54.733579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:54.734546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.735190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.735682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.751558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635d1659-7cf8-4aed-9f61-995ad89719f0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:54.770486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af796e2-a17e-45ee-9a64-a09eda522c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:54.771417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af796e2-a17e-45ee-9a64-a09eda522c9d admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.771708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af796e2-a17e-45ee-9a64-a09eda522c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.771998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af796e2-a17e-45ee-9a64-a09eda522c9d admin admin] PATH_INFO: `/v3/projects/92bb9b5d1e724fb89c90d8923a587912/users/2b938e239f504a91882e6c230cb0e132/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.791853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] MFA Rules not processed for user `ec931c9cc56a462991413926d34dfa46`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:54.794539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:54.818975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1060/2104] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:54.828974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af796e2-a17e-45ee-9a64-a09eda522c9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=92bb9b5d1e724fb89c90d8923a587912, user_id=2b938e239f504a91882e6c230cb0e132, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:54.846111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af796e2-a17e-45ee-9a64-a09eda522c9d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:54.878831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:54.881409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1045/2105] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:11:54 2022] PUT /identity/v3/projects/92bb9b5d1e724fb89c90d8923a587912/users/2b938e239f504a91882e6c230cb0e132/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:54.908175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:54.911229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:54.911923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:54.912560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:54.924205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:54.977161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] MFA Rules not processed for user `2b938e239f504a91882e6c230cb0e132`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:54.980814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.154911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1046/2106] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:55.155953 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:55.163049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.306630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c70a01-24f0-4a81-accc-7a795b7a7997 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:55.313227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:55.313894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:55.314245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:55.315064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] PATH_INFO: `/v3/projects/8034d44cbe8b49c1bd2f32aae4e4807d` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:55.317023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8034d44cbe8b49c1bd2f32aae4e4807d)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:55.319962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:55.348533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8930cc64-6e77-4dca-bc92-df6d430091d5 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:55.407318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8034d44cbe8b49c1bd2f32aae4e4807d', 'request_id': 'req-800002c7-9083-4f39-a45b-de3ff08f05e5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.424712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8034d44cbe8b49c1bd2f32aae4e4807d', 'request_id': 'req-800002c7-9083-4f39-a45b-de3ff08f05e5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.441858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8034d44cbe8b49c1bd2f32aae4e4807d', 'request_id': 'req-800002c7-9083-4f39-a45b-de3ff08f05e5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.473076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] The token cache is being invalidate because project 8034d44cbe8b49c1bd2f32aae4e4807d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:55.473503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.474387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.479058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1047/2107] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:55 2022] DELETE /identity/v3/projects/8034d44cbe8b49c1bd2f32aae4e4807d => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:55.480025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:55.481957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.504906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:55.505769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:55.505769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:55.505769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:55.507989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.522874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:55.567062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800002c7-9083-4f39-a45b-de3ff08f05e5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:55.572173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:55.572878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:55.573154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:55.573425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:55.575302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.618817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:55.620881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:55.633814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1061/2108] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:54 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 757 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:11:55.634817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:55.637431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.655567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cc51ef-7382-4073-a108-b0deb33c86bc service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:55.665659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:55.669445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:55.685983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1048/2109] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:55.700291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:55.732122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:55.734303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:55.804616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:55.805649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:55.806119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:55.806579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] PATH_INFO: `/v3/users/16dcbdd29ad0443a9a4fc731372804df` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:55.808580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] RBAC: Authorizing `identity:delete_user(user_id=16dcbdd29ad0443a9a4fc731372804df)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:55.811867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:55.874299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2e834e-c95e-4c03-8a0d-5d6aa0bd77c4 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:55.880332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:55.881281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:55.881573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:55.881850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] PATH_INFO: `/v3/users/2b938e239f504a91882e6c230cb0e132` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:55.883232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2b938e239f504a91882e6c230cb0e132)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:55.889186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:55.918222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16dcbdd29ad0443a9a4fc731372804df'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.919301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16dcbdd29ad0443a9a4fc731372804df'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.920312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16dcbdd29ad0443a9a4fc731372804df'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.938233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16dcbdd29ad0443a9a4fc731372804df'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.952901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fc654dec-e3d6-47b4-b839-e830fc4c0bcf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16dcbdd29ad0443a9a4fc731372804df'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:55.976082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2b938e239f504a91882e6c230cb0e132'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:56.001960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1062/2110] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:55 2022] DELETE /identity/v3/users/16dcbdd29ad0443a9a4fc731372804df => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:56.003949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2b938e239f504a91882e6c230cb0e132'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:56.025057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2b938e239f504a91882e6c230cb0e132'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:56.026511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2b938e239f504a91882e6c230cb0e132'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:56.028149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2b938e239f504a91882e6c230cb0e132'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:56.069698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:56.092008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1049/2111] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:55 2022] DELETE /identity/v3/users/2b938e239f504a91882e6c230cb0e132 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:11:56.094281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:56.097898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:56.098222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:56.141285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:56.145200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:56.443679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1063/2112] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:56.526543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:56.530677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:56.827818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5ed096-f02b-4546-abb8-d3945e35272b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:56.837581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:56.838919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:56.839943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:56.840647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:56.885650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:56.895817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:56.946475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1050/2113] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:11:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:11:57.232185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:57.239620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:57.326035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590745ed-4892-46dc-8cba-0c69df456853 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:57.351947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:57.354597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:57.355541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:57.355702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] PATH_INFO: `/v3/projects/537b6516b5274c98b1b6255df9c1c3ae` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:57.359924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] RBAC: Authorizing `identity:delete_project(project_id=537b6516b5274c98b1b6255df9c1c3ae)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:57.369924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:57.512894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:57.550766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '537b6516b5274c98b1b6255df9c1c3ae', 'request_id': 'req-aa2b325d-22f3-486a-ada8-79c0fb623487'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:57.575800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '537b6516b5274c98b1b6255df9c1c3ae', 'request_id': 'req-aa2b325d-22f3-486a-ada8-79c0fb623487'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:57.600450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '537b6516b5274c98b1b6255df9c1c3ae', 'request_id': 'req-aa2b325d-22f3-486a-ada8-79c0fb623487'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:57.667456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] The token cache is being invalidate because project 537b6516b5274c98b1b6255df9c1c3ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:57.667968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:57.670730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:57.683999 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1064/2114] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:57 2022] DELETE /identity/v3/projects/537b6516b5274c98b1b6255df9c1c3ae => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:57.685412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:57.821811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0614538f-51c5-4c4a-8c14-dda7c99c18b3 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:57.984398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:57.987796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:58.110948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:58.112917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:58.113459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:58.114010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] PATH_INFO: `/v3/users/eb04301ebf3842c5b0c08cbdc0e5159f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:58.116974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb04301ebf3842c5b0c08cbdc0e5159f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:58.121054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:58.255969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2b325d-22f3-486a-ada8-79c0fb623487 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:58.282384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:58.285044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:58.285577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:58.286111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] PATH_INFO: `/v3/projects/92bb9b5d1e724fb89c90d8923a587912` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:58.289965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=92bb9b5d1e724fb89c90d8923a587912)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:58.298880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb04301ebf3842c5b0c08cbdc0e5159f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.304192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:58.320431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb04301ebf3842c5b0c08cbdc0e5159f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.367074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb04301ebf3842c5b0c08cbdc0e5159f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.368549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb04301ebf3842c5b0c08cbdc0e5159f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.370173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a48d9d13-f58c-4e54-a7e1-d747af395976 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb04301ebf3842c5b0c08cbdc0e5159f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.468532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92bb9b5d1e724fb89c90d8923a587912', 'request_id': 'req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.480996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1051/2115] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:11:57 2022] DELETE /identity/v3/users/eb04301ebf3842c5b0c08cbdc0e5159f => generated 0 bytes in 792 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:58.505145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92bb9b5d1e724fb89c90d8923a587912', 'request_id': 'req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.522868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92bb9b5d1e724fb89c90d8923a587912', 'request_id': 'req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.551089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] The token cache is being invalidate because project 92bb9b5d1e724fb89c90d8923a587912 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:11:58.551835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.555061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-55bcac86-b1fd-40fd-99ca-2c79f8dac2cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:11:58.564015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1065/2116] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:57 2022] DELETE /identity/v3/projects/92bb9b5d1e724fb89c90d8923a587912 => generated 0 bytes in 577 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:11:58.565930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4b637a-939d-4d4c-be25-0754c7b770f1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:58.566378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4b637a-939d-4d4c-be25-0754c7b770f1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:58.567205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4b637a-939d-4d4c-be25-0754c7b770f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:58.633960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:58.634460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:58.634938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:58.663234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-1c4b637a-939d-4d4c-be25-0754c7b770f1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:58.665712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1c4b637a-939d-4d4c-be25-0754c7b770f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:58.689819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] MFA Rules not processed for user `ec931c9cc56a462991413926d34dfa46`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:58.692831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:58.723348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:58.840215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1052/2117] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:58.862940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:58.863441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:58.863915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:58.946156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:11:58.976869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1066/2118] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:11:58.985082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:11:58.990191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:59.005033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1053/2119] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:11:59.009662 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:59.028356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050c6ed0-8a59-4d54-a50e-b0262af6e615 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:59.036134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:59.037526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:59.037989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:59.039252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:59.041627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:11:59.042187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:59.045536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:59.084792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1067/2120] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:11:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:11:59.089980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:59.118046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba6c1f-f571-4d20-b66d-dfbab08cbf25 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:59.143843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:59.144956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:59.147679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:59.148556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:59.150218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:59.157335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:59.319348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1054/2121] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:11:59 2022] POST /identity/v3/projects => generated 359 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:11:59.325717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:59.363181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141ab173-97ef-40c2-abff-5a7a80588076 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:59.627143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:59.628451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:59.628949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:59.629638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:59.630626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:59.630767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:59.634560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:59.636188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:11:59.691555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880afde4-8276-4fab-ac49-fcd8e8e836d7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:59.720534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:59.724130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:59.725180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:59.725335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] PATH_INFO: `/v3/projects/aeab60c25f47457887113c16454e3b0d` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:59.728943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aeab60c25f47457887113c16454e3b0d)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:59.731852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1068/2122] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:11:59 2022] POST /identity/v3/users => generated 280 bytes in 407 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:11:59.736627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:59.738544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:59.819575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b30360-a22d-4575-b5e3-d269082a78e9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:59.845350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:11:59.846029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:11:59.846450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:11:59.846811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:11:59.847744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:11:59.851581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:11:59.873770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1069/2123] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:59 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:11:59.891413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:11:59.954694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9134438a-4141-4fc2-9335-ae8cc9fbecc3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:11:59.979744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aeab60c25f47457887113c16454e3b0d', 'request_id': 'req-db4a7c74-9edd-47ae-9b00-b30da3a10c66'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:00.003036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aeab60c25f47457887113c16454e3b0d', 'request_id': 'req-db4a7c74-9edd-47ae-9b00-b30da3a10c66'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:00.030357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aeab60c25f47457887113c16454e3b0d', 'request_id': 'req-db4a7c74-9edd-47ae-9b00-b30da3a10c66'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:00.078463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] The token cache is being invalidate because project aeab60c25f47457887113c16454e3b0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:12:00.079287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:00.081071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-db4a7c74-9edd-47ae-9b00-b30da3a10c66 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:00.099122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1055/2124] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:11:59 2022] DELETE /identity/v3/projects/aeab60c25f47457887113c16454e3b0d => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:00.106434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb44cfa3-8b14-4d36-99ec-2c6b4f8d3405 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.107508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb44cfa3-8b14-4d36-99ec-2c6b4f8d3405 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.110358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb44cfa3-8b14-4d36-99ec-2c6b4f8d3405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.185094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:00.186077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.186532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.186974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.188064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:00.191519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:00.202721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fb44cfa3-8b14-4d36-99ec-2c6b4f8d3405 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:00.214336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fb44cfa3-8b14-4d36-99ec-2c6b4f8d3405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:00.233904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1070/2125] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:11:59 2022] GET /identity/v3/roles => generated 1420 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:12:00.244741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:00.247569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:00.261634 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1056/2126] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:00.277068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.277595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.277999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.298144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8464bc0-158d-4916-a45d-1325a17c8611 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:00.322201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1933d006-41db-41ed-9e21-0c730fdf418e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:00.323303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1933d006-41db-41ed-9e21-0c730fdf418e admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.323757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1933d006-41db-41ed-9e21-0c730fdf418e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.324223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1933d006-41db-41ed-9e21-0c730fdf418e admin admin] PATH_INFO: `/v3/projects/ab64a228794f45928f40d34372330604/users/a19685b971964e809a2d77bff7901345/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.367276 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:00.381124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:00.383792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1933d006-41db-41ed-9e21-0c730fdf418e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab64a228794f45928f40d34372330604, user_id=a19685b971964e809a2d77bff7901345, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:00.400028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1933d006-41db-41ed-9e21-0c730fdf418e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:00.437758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1057/2127] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:00.446441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:00.498558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1071/2128] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:12:00 2022] PUT /identity/v3/projects/ab64a228794f45928f40d34372330604/users/a19685b971964e809a2d77bff7901345/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:00.508073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4f9c2-3e4f-4a68-8585-26a7e9ca6562 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:00.517807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.518867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.519354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.665706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] MFA Rules not processed for user `a19685b971964e809a2d77bff7901345`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:00.667729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:00.713300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:00.716686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.717159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.717477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.718379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:12:00.718783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:00.721100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:00.729991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:00.741961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:00.769669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1058/2129] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:12:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:12:00.774102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:00.798298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e0dd36-f092-4ace-8f83-e17fa7e72b78 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:00.810090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:00.811075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:00.811482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:00.811772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:00.812457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:00.824127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:00.904307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1072/2130] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:00 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:00.921690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:00.932641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1059/2131] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:12:00 2022] POST /identity/v3/projects => generated 383 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:12:00.951110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a11542-9271-42d6-a171-dc5b6348e37b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:01.000720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:01.003217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:01.310118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f755d66-e597-486c-8ca1-1c0e3727640d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:01.325477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:01.326859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:01.327430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:01.327832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:01.329268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:01.329764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:01.334536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:01.334536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:01.334938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:01.344571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:01.448600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1073/2132] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:12:00 2022] POST /identity/v3/users => generated 292 bytes in 527 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:12:01.452760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:01.482279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1a93d4-7215-42d8-a7d6-860aff20d087 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:01.503524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:01.510621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:01.515610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:01.517363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:01.519101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:01.529309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:01.564279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1074/2133] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:01 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:12:01.571760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:01.583979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:01.586715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:01.626770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f32fc2a-7b0c-48ae-bfc3-a87b413d9454 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:01.634160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1060/2134] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:00 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 634 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:12:01.689360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:01.710822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:01.712073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:01.712609 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:01.713114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:01.719527 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:01.727498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:01.770068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2732e906-774d-4993-b63d-9c99754239ff service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:01.784384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:01.787799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1075/2135] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:01 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:12:01.790038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:01.791581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:01.792206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] PATH_INFO: `/v3/users/a19685b971964e809a2d77bff7901345` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:01.795156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a19685b971964e809a2d77bff7901345)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:01.799381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:01.800434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:01.887559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592b6d07-4f71-4adf-a080-c25c52abc181 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:01.911212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee40a9d-2048-4092-8b9c-4db653aa2359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:01.914980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee40a9d-2048-4092-8b9c-4db653aa2359 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:01.917762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee40a9d-2048-4092-8b9c-4db653aa2359 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:01.918247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee40a9d-2048-4092-8b9c-4db653aa2359 admin admin] PATH_INFO: `/v3/projects/05d389d2a5e840a0b26307bdf1168ed1/users/3e937cb2f0e34b4eae3e5d2c11170edb/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:02.019602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a19685b971964e809a2d77bff7901345'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:02.026388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee40a9d-2048-4092-8b9c-4db653aa2359 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05d389d2a5e840a0b26307bdf1168ed1, user_id=3e937cb2f0e34b4eae3e5d2c11170edb, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:02.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee40a9d-2048-4092-8b9c-4db653aa2359 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:02.047761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a19685b971964e809a2d77bff7901345'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:02.066133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a19685b971964e809a2d77bff7901345'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:02.067450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a19685b971964e809a2d77bff7901345'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:02.070815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22311ce8-071d-4c64-8caf-ab46be51b196 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a19685b971964e809a2d77bff7901345'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:02.079299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1076/2136] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:12:01 2022] PUT /identity/v3/projects/05d389d2a5e840a0b26307bdf1168ed1/users/3e937cb2f0e34b4eae3e5d2c11170edb/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:02.103215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:02.103896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:02.104611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:02.123659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:02.146224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1061/2137] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:12:01 2022] DELETE /identity/v3/users/a19685b971964e809a2d77bff7901345 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:02.153444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:02.158116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:02.160101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:02.243489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:02.245074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:02.273863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] MFA Rules not processed for user `3e937cb2f0e34b4eae3e5d2c11170edb`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:02.288799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:02.519410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:02.528530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1062/2138] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:02.552965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1077/2139] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 452 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:02.565865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:02.569018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:02.583751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:02.588591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:02.912984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ae9ac0-34b4-4fa8-b0f5-a090fba5f4b0 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:02.927634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:02.928759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:02.929502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:02.930136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:02.961640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:02.964537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:02.988999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d3bef-708a-48fe-bfe0-6975552a1a73 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:03.009463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1713b414-d4e4-4dc1-849c-d93805c310ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:03.010482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1713b414-d4e4-4dc1-849c-d93805c310ab service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:03.010948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1713b414-d4e4-4dc1-849c-d93805c310ab service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:03.012947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1078/2140] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:12:03.013392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1713b414-d4e4-4dc1-849c-d93805c310ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:03.055781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1713b414-d4e4-4dc1-849c-d93805c310ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:03.057819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1713b414-d4e4-4dc1-849c-d93805c310ab service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:03.136441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1063/2141] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:12:04.206261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:04.218528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:04.261358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c50f5-e04c-4b19-8386-1ba040b3ecdc service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:04.275407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:04.276513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:04.277493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:04.278120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] PATH_INFO: `/v3/projects/ab64a228794f45928f40d34372330604` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:04.285692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab64a228794f45928f40d34372330604)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:04.293229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:04.491601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab64a228794f45928f40d34372330604', 'request_id': 'req-93a1edb9-1955-4bb2-a6c5-954ce3407caa'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:04.531055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab64a228794f45928f40d34372330604', 'request_id': 'req-93a1edb9-1955-4bb2-a6c5-954ce3407caa'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:04.601774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab64a228794f45928f40d34372330604', 'request_id': 'req-93a1edb9-1955-4bb2-a6c5-954ce3407caa'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:04.676160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] The token cache is being invalidate because project ab64a228794f45928f40d34372330604 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:12:04.676694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:04.678673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-93a1edb9-1955-4bb2-a6c5-954ce3407caa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:04.679208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1079/2142] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:12:04 2022] DELETE /identity/v3/projects/ab64a228794f45928f40d34372330604 => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:12:04.686095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:04.687432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:04.687918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:04.722288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:04.724494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:04.865102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1064/2143] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:04.874389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:04.874789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:04.875252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:04.917130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:04.918513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:04.938506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1080/2144] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:04.944924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:04.966704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c71f7-d71a-4dd7-b592-6cc1f5f109c7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:04.974514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:04.976173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:04.976509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:04.976788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:04.977620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:12:04.981152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:04.984057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:05.026331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:05.026838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1065/2145] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:12:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:12:05.055138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc6794d-67e4-4019-94f2-2fd7255e8a52 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:05.064897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:05.066023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:05.066590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:05.067099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:05.068631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:05.073765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:05.139647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1081/2146] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:12:05 2022] POST /identity/v3/projects => generated 373 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:12:05.146814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:05.161538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24e560e-a7d6-4994-b1e4-8062ffa53480 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:05.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:05.341131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:05.341928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:05.342456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:05.342786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:05.343529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:05.345676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:05.403242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1066/2147] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:12:05 2022] POST /identity/v3/users => generated 287 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:12:05.405263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:05.434602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70008d-81c0-4d02-adc7-29c7110a4bde admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:05.442698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:05.443652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:05.444122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:05.444670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:05.453284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:05.459819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:05.506666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1082/2148] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:05 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:12:05.513656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:05.554668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85c0bbb-6650-4038-b6ef-f617c0420a91 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:05.577259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19631eb9-bbe6-4404-8155-125f8f00d855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:05.578082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19631eb9-bbe6-4404-8155-125f8f00d855 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:05.578380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19631eb9-bbe6-4404-8155-125f8f00d855 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:05.578661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19631eb9-bbe6-4404-8155-125f8f00d855 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:05.582060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19631eb9-bbe6-4404-8155-125f8f00d855 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:05.588448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19631eb9-bbe6-4404-8155-125f8f00d855 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:05.628819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1067/2149] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:05 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:12:05.634673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:05.649226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92057ae3-6779-497c-aab7-453245f930f9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:05.681222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:05.681222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:05.681222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:05.681222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] PATH_INFO: `/v3/projects/090baf17bbce4182a0a6c6aa06113ce6/users/4b8557e6fe894521b9eb9e50bd92ecd6/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:05.747212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=090baf17bbce4182a0a6c6aa06113ce6, user_id=4b8557e6fe894521b9eb9e50bd92ecd6, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:05.750027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:05.793515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1083/2150] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:12:05 2022] PUT /identity/v3/projects/090baf17bbce4182a0a6c6aa06113ce6/users/4b8557e6fe894521b9eb9e50bd92ecd6/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:12:05.795292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:05.796024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:05.796024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:05.851043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] MFA Rules not processed for user `4b8557e6fe894521b9eb9e50bd92ecd6`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:05.853006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:06.076403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1068/2151] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:06.203097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:06.207967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:06.265747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:06.335807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540646b5-056b-4a61-9a10-4fb253d31fbe admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:06.342003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc053ba6-c483-4aef-926e-96733d7a26b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:06.343209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc053ba6-c483-4aef-926e-96733d7a26b1 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:06.343786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc053ba6-c483-4aef-926e-96733d7a26b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:06.344385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc053ba6-c483-4aef-926e-96733d7a26b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:06.365644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc053ba6-c483-4aef-926e-96733d7a26b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:06.367860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc053ba6-c483-4aef-926e-96733d7a26b1 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:06.391496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1084/2152] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:06 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:12:06.477608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:06.479545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:06.659930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9609a3c4-c4d6-4160-be1b-a0ea045d5ba4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:06.691064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:06.691832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:06.692456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:06.692777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] PATH_INFO: `/v3/users/3e937cb2f0e34b4eae3e5d2c11170edb` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:06.697458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e937cb2f0e34b4eae3e5d2c11170edb)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:06.699887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:06.815513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:06.834061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e937cb2f0e34b4eae3e5d2c11170edb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:06.856793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e937cb2f0e34b4eae3e5d2c11170edb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:06.898152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e937cb2f0e34b4eae3e5d2c11170edb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:06.900536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e937cb2f0e34b4eae3e5d2c11170edb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:06.901872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e937cb2f0e34b4eae3e5d2c11170edb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:06.975853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1069/2153] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:12:06 2022] DELETE /identity/v3/users/3e937cb2f0e34b4eae3e5d2c11170edb => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:06.982475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:06.983091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:06.983091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:07.071447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:07.080377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:07.334608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1085/2154] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:07.458846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:07.459301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:07.702223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed3ee24-6cb6-40cd-b924-1b58d8c2b5df admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:07.710621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329e4372-11f2-4145-991d-a579b1d7ebca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:07.710621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e4372-11f2-4145-991d-a579b1d7ebca service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:07.724626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e4372-11f2-4145-991d-a579b1d7ebca service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:07.725225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e4372-11f2-4145-991d-a579b1d7ebca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:07.797167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329e4372-11f2-4145-991d-a579b1d7ebca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:07.803796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329e4372-11f2-4145-991d-a579b1d7ebca service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:07.851124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1070/2155] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:12:08.942176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:08.943868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:08.982198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69e5432-44c4-4bb6-9737-838ed5e09085 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:08.994349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:08.995389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:08.995910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:08.996700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] PATH_INFO: `/v3/projects/05d389d2a5e840a0b26307bdf1168ed1` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:08.998935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] RBAC: Authorizing `identity:delete_project(project_id=05d389d2a5e840a0b26307bdf1168ed1)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:09.003149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:09.139148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05d389d2a5e840a0b26307bdf1168ed1', 'request_id': 'req-2af2e4f2-9720-457c-9502-2d463162fabf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:09.159495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05d389d2a5e840a0b26307bdf1168ed1', 'request_id': 'req-2af2e4f2-9720-457c-9502-2d463162fabf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:09.173926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05d389d2a5e840a0b26307bdf1168ed1', 'request_id': 'req-2af2e4f2-9720-457c-9502-2d463162fabf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:09.202812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] The token cache is being invalidate because project 05d389d2a5e840a0b26307bdf1168ed1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:12:09.203142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:09.205110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2af2e4f2-9720-457c-9502-2d463162fabf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:09.210086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1086/2156] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:12:08 2022] DELETE /identity/v3/projects/05d389d2a5e840a0b26307bdf1168ed1 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:09.218429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb4e799-a23d-4396-8a3e-6f6284dad1f6 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:09.220810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb4e799-a23d-4396-8a3e-6f6284dad1f6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:09.221909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb4e799-a23d-4396-8a3e-6f6284dad1f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:09.350411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8fb4e799-a23d-4396-8a3e-6f6284dad1f6 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:09.355648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8fb4e799-a23d-4396-8a3e-6f6284dad1f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:09.531744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:09.534925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:09.536049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:09.660211 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1071/2157] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:09.664322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:09.664807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:09.665199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:09.665930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] MFA Rules not processed for user `e65f0f1ba54f483abff788974b1256cb`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:09.680437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:09.730194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:09.731554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:09.752928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1072/2158] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:09.759822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:09.778938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53bc0036-e7ba-46b8-ace3-35b44377e2e8 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:09.790479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:09.791434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:09.791866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:09.792358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:09.793300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:12:09.793840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:09.796765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:09.799983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:09.824576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1087/2159] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:09.833924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1073/2160] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:12:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:12:09.842257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:09.860827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696635cd-28b3-4e9c-8974-324f219a3038 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:09.871131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:09.872310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:09.872951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:09.873584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:09.874845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:09.878062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:09.936405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1088/2161] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:12:09 2022] POST /identity/v3/projects => generated 375 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:12:09.938761 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:09.956133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db024dea-541f-4434-a650-3d509bc34eea admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:10.051605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:10.102455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:10.103782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:10.104394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:10.104968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:10.106361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:10.110510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:10.152336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1074/2162] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:12:09 2022] POST /identity/v3/users => generated 288 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:12:10.158320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:10.177747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b6c8d1-2903-40dc-94d7-53c4d6492d89 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:10.185434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:10.187226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:10.188114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:10.188952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:10.191712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:10.196419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:10.222902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1089/2163] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:12:10.228327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:10.258587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b2d6e2-dd0d-42c4-9789-5cb2efd544db admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:10.267685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d2671a-c733-4358-be2f-041eb0c315ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:10.269964 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d2671a-c733-4358-be2f-041eb0c315ef admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:10.270219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d2671a-c733-4358-be2f-041eb0c315ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:10.270553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d2671a-c733-4358-be2f-041eb0c315ef admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:10.271890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d2671a-c733-4358-be2f-041eb0c315ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:10.274346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d2671a-c733-4358-be2f-041eb0c315ef admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:10.310372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1075/2164] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:12:10.315738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:10.331314 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b34ba7-f5e9-4dc2-a1a3-482834968fd9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:10.339014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:10.340106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:10.340444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:10.340793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] PATH_INFO: `/v3/projects/4203fc81bbd14439bb6ce329a5ccbc79/users/0ab3ed0f7df742618c77815ef1ddb960/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:10.363777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4203fc81bbd14439bb6ce329a5ccbc79, user_id=0ab3ed0f7df742618c77815ef1ddb960, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:10.369668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:10.392619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1090/2165] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:12:10 2022] PUT /identity/v3/projects/4203fc81bbd14439bb6ce329a5ccbc79/users/0ab3ed0f7df742618c77815ef1ddb960/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:10.411119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc80112-dd0c-42c8-b7fa-55672df28db6 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:10.411621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc80112-dd0c-42c8-b7fa-55672df28db6 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:10.412115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc80112-dd0c-42c8-b7fa-55672df28db6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:10.469866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6bc80112-dd0c-42c8-b7fa-55672df28db6 None None] MFA Rules not processed for user `0ab3ed0f7df742618c77815ef1ddb960`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:10.472293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6bc80112-dd0c-42c8-b7fa-55672df28db6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:10.615091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1076/2166] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:10 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:10.647160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:10.649867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:10.761560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e1b28-16e6-4c11-970f-789eaf226d9b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:10.766691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:10.766913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:10.767176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:10.767412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:10.783725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:10.785294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:10.800529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1091/2167] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:12:12.543452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7b47e7-e0dc-4423-9a74-d1ce3652642d None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:12.544670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7b47e7-e0dc-4423-9a74-d1ce3652642d None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:12.545198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7b47e7-e0dc-4423-9a74-d1ce3652642d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:12.569392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7f7b47e7-e0dc-4423-9a74-d1ce3652642d None None] MFA Rules not processed for user `0ab3ed0f7df742618c77815ef1ddb960`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:12.570754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f7b47e7-e0dc-4423-9a74-d1ce3652642d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:12.580558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1077/2168] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:17.876030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:17.879286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:18.076568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a744e8f9-e68b-4b22-bc5e-18f418b4c466 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:18.092665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:18.095780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:18.098848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:18.099978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] PATH_INFO: `/v3/users/806634a2bab64f9da24dda8048c73c3a` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:18.103579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=806634a2bab64f9da24dda8048c73c3a)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:18.108187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:18.276904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:18.295970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '806634a2bab64f9da24dda8048c73c3a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:18.298096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '806634a2bab64f9da24dda8048c73c3a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:18.299464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '806634a2bab64f9da24dda8048c73c3a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:18.316121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '806634a2bab64f9da24dda8048c73c3a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:18.338883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '806634a2bab64f9da24dda8048c73c3a'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:18.368906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1092/2169] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:12:17 2022] DELETE /identity/v3/users/806634a2bab64f9da24dda8048c73c3a => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:18.374593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:18.375413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:18.375567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:18.398287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:18.399546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:18.511175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1078/2170] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:18.525052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:18.527484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:18.622481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efdb943-5c6e-4fb1-acd9-b3753315bc72 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:18.626723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7fa045-a88e-4e39-9c3a-214cfa7c68ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:18.628046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fa045-a88e-4e39-9c3a-214cfa7c68ee service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:18.628457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fa045-a88e-4e39-9c3a-214cfa7c68ee service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:18.628823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fa045-a88e-4e39-9c3a-214cfa7c68ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:18.640771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7fa045-a88e-4e39-9c3a-214cfa7c68ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:18.642315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7fa045-a88e-4e39-9c3a-214cfa7c68ee service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:18.654398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1093/2171] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:12:19.281100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:19.283381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:19.309108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aae521c-219f-4645-9b5e-fbb09f355bb5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:19.336021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:19.337185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:19.337657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:19.338108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] PATH_INFO: `/v3/projects/513191b6f2a54b479687e09408c1989d` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:19.341709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] RBAC: Authorizing `identity:delete_project(project_id=513191b6f2a54b479687e09408c1989d)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:19.347063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:19.430701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:19.450383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '513191b6f2a54b479687e09408c1989d', 'request_id': 'req-e6f7065a-91a0-4fbd-a07a-5c5baa503078'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:19.471540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '513191b6f2a54b479687e09408c1989d', 'request_id': 'req-e6f7065a-91a0-4fbd-a07a-5c5baa503078'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:19.490225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '513191b6f2a54b479687e09408c1989d', 'request_id': 'req-e6f7065a-91a0-4fbd-a07a-5c5baa503078'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:19.506251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] The token cache is being invalidate because project 513191b6f2a54b479687e09408c1989d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:12:19.506555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:19.507339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e6f7065a-91a0-4fbd-a07a-5c5baa503078 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:19.509002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1079/2172] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:12:19 2022] DELETE /identity/v3/projects/513191b6f2a54b479687e09408c1989d => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:19.516931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:19.517840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:19.518125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:19.554316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:19.556673 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:19.660376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1094/2173] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:19.673187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:19.673674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:19.674188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:19.714778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:19.716744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:19.743064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1080/2174] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:19.758129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:19.777465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a9a786-582d-4aa8-8d18-6dfcf2376872 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:19.786812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:19.787916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:19.788448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:19.788964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:19.790181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:12:19.790739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:19.794032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:19.817747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1095/2175] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:12:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:12:19.824056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:19.843302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46f894e-f1f7-4b4d-8791-8f57d3736c9b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:19.852732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:19.853737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:19.854202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:19.854671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:19.855910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:19.859840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:19.910161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1081/2176] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:12:19 2022] POST /identity/v3/projects => generated 365 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:12:19.917604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:19.935954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753e1088-4a91-4f9c-89ee-3d2e1311228b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:20.017823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:20.043364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:20.044563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:20.044975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:20.045555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:20.046850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:20.049748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:20.082825 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1096/2177] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:12:19 2022] POST /identity/v3/users => generated 283 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:12:20.087599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:20.105093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64de5180-f428-49eb-b365-ddd11feb476d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:20.111867 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:20.112731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:20.113104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:20.113490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:20.114280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:20.116112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:20.132572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1082/2178] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:12:20.137888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:20.156965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b2bb76-614f-4b7f-b704-15d1546e478c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:20.165821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc5b1f-b40f-404d-a323-8d3d507eec7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:20.166891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc5b1f-b40f-404d-a323-8d3d507eec7c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:20.167490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc5b1f-b40f-404d-a323-8d3d507eec7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:20.167883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc5b1f-b40f-404d-a323-8d3d507eec7c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:20.169063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc5b1f-b40f-404d-a323-8d3d507eec7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:20.171994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc5b1f-b40f-404d-a323-8d3d507eec7c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:20.204816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1097/2179] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:12:20.210806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:20.232694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc5196f-448e-4e4e-aca9-26fb149ba734 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:20.242848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:20.244271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:20.246708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:20.246708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] PATH_INFO: `/v3/projects/c904b341e34c4b64a2b4eb8772d19279/users/7995c2d62dee44d2bf35dc22a08d1afb/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:20.276081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c904b341e34c4b64a2b4eb8772d19279, user_id=7995c2d62dee44d2bf35dc22a08d1afb, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:20.282055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:20.301797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1083/2180] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:12:20 2022] PUT /identity/v3/projects/c904b341e34c4b64a2b4eb8772d19279/users/7995c2d62dee44d2bf35dc22a08d1afb/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:20.312674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:20.313042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:20.313318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:20.364105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] MFA Rules not processed for user `7995c2d62dee44d2bf35dc22a08d1afb`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:20.366782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:20.499501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1098/2181] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:20 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:12:20.517752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:20.521517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:20.629324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc99c88-3c0c-4afd-95db-214f28c7721f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:20.633894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24643eb5-b3f2-4ffb-9784-3dddb61e7126 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:20.634610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24643eb5-b3f2-4ffb-9784-3dddb61e7126 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:20.634936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24643eb5-b3f2-4ffb-9784-3dddb61e7126 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:20.635218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24643eb5-b3f2-4ffb-9784-3dddb61e7126 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:20.648983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24643eb5-b3f2-4ffb-9784-3dddb61e7126 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:20.652784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24643eb5-b3f2-4ffb-9784-3dddb61e7126 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:20.671791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1084/2182] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:20 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:12:22.097196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:22.099731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:22.194266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695e2f20-0e58-4b8d-bb81-33fb491ae529 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:22.199193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:22.199860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:22.200141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:22.200562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] PATH_INFO: `/v3/users/4b8557e6fe894521b9eb9e50bd92ecd6` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:22.202187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b8557e6fe894521b9eb9e50bd92ecd6)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:22.204478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:22.302317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b8557e6fe894521b9eb9e50bd92ecd6'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:22.303058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b8557e6fe894521b9eb9e50bd92ecd6'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:22.303759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b8557e6fe894521b9eb9e50bd92ecd6'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:22.319897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b8557e6fe894521b9eb9e50bd92ecd6'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:22.327696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:22.333349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b8557e6fe894521b9eb9e50bd92ecd6'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:22.376809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1099/2183] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:12:22 2022] DELETE /identity/v3/users/4b8557e6fe894521b9eb9e50bd92ecd6 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:22.384334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:22.385089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:22.385238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:22.430740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:22.432843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:22.552144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1085/2184] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:12:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:22.564515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:22.567621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:22.702432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd709e4-133c-4ae1-90ab-b046c9c0efac admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:22.707587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5437ac-623a-43bf-98fa-70a9780b36b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:22.708406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5437ac-623a-43bf-98fa-70a9780b36b6 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:22.708717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5437ac-623a-43bf-98fa-70a9780b36b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:22.709141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5437ac-623a-43bf-98fa-70a9780b36b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:22.726348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5437ac-623a-43bf-98fa-70a9780b36b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:22.727829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5437ac-623a-43bf-98fa-70a9780b36b6 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:22.758329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1100/2185] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:12:24.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:24.670936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:24.696719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:24.706278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc65c1f-bfa7-420f-bc11-cea3b08d93d5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:24.724842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:24.736684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:24.739028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:24.740690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] PATH_INFO: `/v3/projects/090baf17bbce4182a0a6c6aa06113ce6` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:24.743212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] RBAC: Authorizing `identity:delete_project(project_id=090baf17bbce4182a0a6c6aa06113ce6)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:24.747564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:24.814574 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '090baf17bbce4182a0a6c6aa06113ce6', 'request_id': 'req-45a2798d-45d6-43cd-96b3-66d1f1e3418f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:24.831288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '090baf17bbce4182a0a6c6aa06113ce6', 'request_id': 'req-45a2798d-45d6-43cd-96b3-66d1f1e3418f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:24.848586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '090baf17bbce4182a0a6c6aa06113ce6', 'request_id': 'req-45a2798d-45d6-43cd-96b3-66d1f1e3418f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:24.874984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] The token cache is being invalidate because project 090baf17bbce4182a0a6c6aa06113ce6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:12:24.875250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:24.875968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:12:24.880862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1086/2186] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:12:24 2022] DELETE /identity/v3/projects/090baf17bbce4182a0a6c6aa06113ce6 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:12:27.037900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce5357-d9f7-4429-a2df-d00916189b26 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:27.038859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce5357-d9f7-4429-a2df-d00916189b26 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:27.039354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce5357-d9f7-4429-a2df-d00916189b26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:27.077921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-09ce5357-d9f7-4429-a2df-d00916189b26 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:27.077921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-09ce5357-d9f7-4429-a2df-d00916189b26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:27.221821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1101/2187] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:12:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:27.352660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:27.356026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:27.615792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a2798d-45d6-43cd-96b3-66d1f1e3418f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:27.644577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:27.656307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:27.656307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:27.656307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:27.692161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:27.695303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:27.735815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1087/2188] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:12:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:12:36.490707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:36.492774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:36.494562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:36.558559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:12:36.562099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:12:36.585782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1102/2189] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:12:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:12:36.631295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:36.653575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e98ede-bc44-4671-89bf-e3098fd19c5d service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:12:36.668055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866185e7-2e9e-4529-9083-e9f3b7b21a76 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:12:36.669203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866185e7-2e9e-4529-9083-e9f3b7b21a76 service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:12:36.669715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866185e7-2e9e-4529-9083-e9f3b7b21a76 service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:12:36.670206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866185e7-2e9e-4529-9083-e9f3b7b21a76 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:12:36.695654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866185e7-2e9e-4529-9083-e9f3b7b21a76 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:12:36.697712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866185e7-2e9e-4529-9083-e9f3b7b21a76 service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:12:36.720937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1088/2190] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:12:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:12:46.542338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=63,keystone:DELETE=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:46.781706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=63,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:12:59.541924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:12:59.544982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:00.090193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1ddbc9-0315-4abd-9111-5e82fa5769f2 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:00.114181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:00.115640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:00.116158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:00.116778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:00.122652 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:00.166363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:00.168995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bc42b8-125c-4f35-92ff-03aa629e1790 service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:00.220931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1103/2191] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:12:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 680 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:03.865349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a800b296-d327-4406-bcea-63d7adecbf7a None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:03.865998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a800b296-d327-4406-bcea-63d7adecbf7a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:03.865998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a800b296-d327-4406-bcea-63d7adecbf7a None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:03.867902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1089/2192] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:13:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:13:03.876000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c0a1b-785b-4485-81f1-093ba21d3fa3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:03.876766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c0a1b-785b-4485-81f1-093ba21d3fa3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:03.877055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c0a1b-785b-4485-81f1-093ba21d3fa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:03.905552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5e8c0a1b-785b-4485-81f1-093ba21d3fa3 None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:03.907249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5e8c0a1b-785b-4485-81f1-093ba21d3fa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:04.065781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1104/2193] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:13:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:07.357703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a800b296-d327-4406-bcea-63d7adecbf7a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:07.360902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a800b296-d327-4406-bcea-63d7adecbf7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:07.444658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a800b296-d327-4406-bcea-63d7adecbf7a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:07.458257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:07.459212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:07.459718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:07.460285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:07.485082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:07.487053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:07.501469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1090/2194] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:13:07.664017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab41b7a-56bd-400c-9f05-c3abcba3cf84 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:07.665284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab41b7a-56bd-400c-9f05-c3abcba3cf84 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:07.665284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab41b7a-56bd-400c-9f05-c3abcba3cf84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:07.706366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6ab41b7a-56bd-400c-9f05-c3abcba3cf84 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:07.717947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6ab41b7a-56bd-400c-9f05-c3abcba3cf84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:07.729869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1105/2195] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:13:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:07.763076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:07.783061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d86ef-c446-40f3-a760-ed00a363c6ec service glance] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:07.816499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4218a62-97d4-49cd-8210-a38d1c152b15 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:07.816499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4218a62-97d4-49cd-8210-a38d1c152b15 service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:07.816499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4218a62-97d4-49cd-8210-a38d1c152b15 service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:07.816499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4218a62-97d4-49cd-8210-a38d1c152b15 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:07.827796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4218a62-97d4-49cd-8210-a38d1c152b15 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:07.832354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4218a62-97d4-49cd-8210-a38d1c152b15 service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:07.848514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1091/2196] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 07 01:13:08.115169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7e75a6-53d8-410c-9b98-febcd60bff0f None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.115488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7e75a6-53d8-410c-9b98-febcd60bff0f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.115830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7e75a6-53d8-410c-9b98-febcd60bff0f None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.118580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1106/2197] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:13:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:13:08.125938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8a7e1a-86c2-4c2f-8295-b932795d1a6e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.126222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8a7e1a-86c2-4c2f-8295-b932795d1a6e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.126525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8a7e1a-86c2-4c2f-8295-b932795d1a6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.163782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-dd8a7e1a-86c2-4c2f-8295-b932795d1a6e None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:08.165386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dd8a7e1a-86c2-4c2f-8295-b932795d1a6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:08.240606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1092/2198] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:13:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:08.257986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7e75a6-53d8-410c-9b98-febcd60bff0f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:08.261065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-df7e75a6-53d8-410c-9b98-febcd60bff0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:08.290772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7e75a6-53d8-410c-9b98-febcd60bff0f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:08.296499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae518bf-1793-4367-9863-e85c55cb9f42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:08.299718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae518bf-1793-4367-9863-e85c55cb9f42 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.300009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae518bf-1793-4367-9863-e85c55cb9f42 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.301071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae518bf-1793-4367-9863-e85c55cb9f42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.312523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae518bf-1793-4367-9863-e85c55cb9f42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:08.314450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae518bf-1793-4367-9863-e85c55cb9f42 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:08.336335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1107/2199] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:13:08.441658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8dc3a-7f51-4778-afb1-4a6908d6bf61 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.442475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8dc3a-7f51-4778-afb1-4a6908d6bf61 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.442743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8dc3a-7f51-4778-afb1-4a6908d6bf61 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.444528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1093/2200] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:13:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:13:08.451824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823074c3-c51c-4c24-8eed-c4d77841532f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.452114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823074c3-c51c-4c24-8eed-c4d77841532f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.452436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823074c3-c51c-4c24-8eed-c4d77841532f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.476079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-823074c3-c51c-4c24-8eed-c4d77841532f None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:08.477434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-823074c3-c51c-4c24-8eed-c4d77841532f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:08.489989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1108/2201] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:13:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:08.506678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8dc3a-7f51-4778-afb1-4a6908d6bf61 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:08.527423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8dc3a-7f51-4778-afb1-4a6908d6bf61 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:08.534834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:08.535510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.535910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.536193 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.551591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:08.553532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:08.569890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1094/2202] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:08.666170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201390c-eef6-4be3-b999-463ec7b31542 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.667179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201390c-eef6-4be3-b999-463ec7b31542 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.667179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201390c-eef6-4be3-b999-463ec7b31542 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7201390c-eef6-4be3-b999-463ec7b31542 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:08.694072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7201390c-eef6-4be3-b999-463ec7b31542 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:08.704204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1109/2203] 10.222.0.26 () {58 vars in 967 bytes} [Sun Aug 7 01:13:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:08.717954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:08.734732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f808ce5-57ac-45bc-a6be-294fe9ae3ed1 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:08.744172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:08.745256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:08.745753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:08.746217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:08.760139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:08.762894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:08.779905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1095/2204] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:13:09.143545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a5670-4c02-4736-823b-54ee4307ef5c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:09.143835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a5670-4c02-4736-823b-54ee4307ef5c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:09.144211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a5670-4c02-4736-823b-54ee4307ef5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:09.177098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f43a5670-4c02-4736-823b-54ee4307ef5c None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:09.178771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f43a5670-4c02-4736-823b-54ee4307ef5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:09.190706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1110/2205] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:13:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:09.214338 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:09.230985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cdb3e0-8695-49d7-aa88-b34cb1eb0af4 service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:09.253839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:09.254476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:09.254782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:09.255063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:09.292808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:09.312494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:09.332038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1096/2206] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 07 01:13:09.528877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfea1c3-22c7-4588-92f3-4d4ae3adba57 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:09.529171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfea1c3-22c7-4588-92f3-4d4ae3adba57 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:09.530921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfea1c3-22c7-4588-92f3-4d4ae3adba57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:09.557152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-abfea1c3-22c7-4588-92f3-4d4ae3adba57 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:09.558727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-abfea1c3-22c7-4588-92f3-4d4ae3adba57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:09.568320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1111/2207] 10.222.0.26 () {58 vars in 967 bytes} [Sun Aug 7 01:13:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:09.586562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:09.607633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931ac468-d6f6-471d-9c2f-7be9c96620e2 service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:09.621132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:09.622189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:09.623781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:09.623781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:09.639678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:09.641758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:09.654202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1097/2208] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:13:09.834357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:09.834828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:09.835149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:09.861003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:09.862697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:09.870891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1112/2209] 10.222.0.26 () {58 vars in 967 bytes} [Sun Aug 7 01:13:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:09.885720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:09.899524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8d9db6-f2ce-481b-952b-2dfd4319540b service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:09.907163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f9f73-5e3e-4f6e-94ac-2362efd54e09 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:09.908419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f9f73-5e3e-4f6e-94ac-2362efd54e09 service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:09.909296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f9f73-5e3e-4f6e-94ac-2362efd54e09 service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:09.910023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f9f73-5e3e-4f6e-94ac-2362efd54e09 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:09.928385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648f9f73-5e3e-4f6e-94ac-2362efd54e09 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:09.931303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648f9f73-5e3e-4f6e-94ac-2362efd54e09 service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:09.947469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1098/2210] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:13:12.297398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:12.310784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:12.323405 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:12.324043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:12.324043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:12.349728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:12.352083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:12.365024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:12.368077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1099/2211] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:13:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:12.384376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:12.407133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36a8c6-00b2-4471-a16a-28a8a7376fd9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:12.416414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8e171b-a424-4c81-81da-2cdf5da13214 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:12.417470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:12.419836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:12.419836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:12.419836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:12.426363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:12.427895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:12.428207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:12.428742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] PATH_INFO: `/v3/users/dfc9712739724c91bd9dd5fdaaebd642` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:12.432383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dfc9712739724c91bd9dd5fdaaebd642)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:12.433446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:12.435345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:12.437430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:12.458887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1100/2212] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:13:12.533863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dfc9712739724c91bd9dd5fdaaebd642'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:12.534898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dfc9712739724c91bd9dd5fdaaebd642'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:12.535838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dfc9712739724c91bd9dd5fdaaebd642'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:12.552417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dfc9712739724c91bd9dd5fdaaebd642'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:12.568573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dfc9712739724c91bd9dd5fdaaebd642'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:12.609667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1113/2213] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:13:12 2022] DELETE /identity/v3/users/dfc9712739724c91bd9dd5fdaaebd642 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:12.903022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:12.904799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:12.927264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e4ad2e-2df2-4206-8089-b7340035fdef service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:12.965027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:13.107012 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:13.108617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:13.129083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:13.130638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:13.131481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:13.132106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] PATH_INFO: `/v3/users/0ab3ed0f7df742618c77815ef1ddb960` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:13.136271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ab3ed0f7df742618c77815ef1ddb960)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:13.141582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:13.285133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ab3ed0f7df742618c77815ef1ddb960'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.316956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ab3ed0f7df742618c77815ef1ddb960'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.326914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e390e3-733b-4338-805c-799e39219213 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:13.335080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:13.337175 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:13.337792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:13.338560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] PATH_INFO: `/v3/projects/1c3de7a94e61473aa759f03c19119775` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:13.340708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c3de7a94e61473aa759f03c19119775)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:13.341964 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ab3ed0f7df742618c77815ef1ddb960'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.344980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ab3ed0f7df742618c77815ef1ddb960'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.345956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:13.350528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2a393cb1-b59e-4ae7-a25b-5b20ebeaad8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ab3ed0f7df742618c77815ef1ddb960'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.452425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1101/2214] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:13:12 2022] DELETE /identity/v3/users/0ab3ed0f7df742618c77815ef1ddb960 => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:13.455731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8c8083-4a1c-48a9-828a-ecdb64894834 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:13.468094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8c8083-4a1c-48a9-828a-ecdb64894834 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:13.469208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8c8083-4a1c-48a9-828a-ecdb64894834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:13.494549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c3de7a94e61473aa759f03c19119775', 'request_id': 'req-625c8874-6f7a-4f84-9ea5-4304f3551696'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.499755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:13.526887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c3de7a94e61473aa759f03c19119775', 'request_id': 'req-625c8874-6f7a-4f84-9ea5-4304f3551696'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.558445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c3de7a94e61473aa759f03c19119775', 'request_id': 'req-625c8874-6f7a-4f84-9ea5-4304f3551696'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.595341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] The token cache is being invalidate because project 1c3de7a94e61473aa759f03c19119775 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:13:13.597705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.598802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-625c8874-6f7a-4f84-9ea5-4304f3551696 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:13.601735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1114/2215] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:13:13 2022] DELETE /identity/v3/projects/1c3de7a94e61473aa759f03c19119775 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:13.620027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-fa8c8083-4a1c-48a9-828a-ecdb64894834 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:13.623378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fa8c8083-4a1c-48a9-828a-ecdb64894834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:13.630354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:13.630637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:13.631344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:13.694355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:13.697207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:13.903607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1102/2216] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:13.904679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1115/2217] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:13.922912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:13.923604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:13.924201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:13.929719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:13.931945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:14.011461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:14.014577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:14.030360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1103/2218] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:14.037563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.075410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bf705-fefc-4098-a34f-0cc9fd758435 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.079853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa6937a-06f2-45ed-bfd6-500b888ddcd7 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.087688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.088640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.088852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.089460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.098138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.100652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.100991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.101266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.103689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:13:14.103994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.120681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.140115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.152685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.153358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1104/2219] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:13:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:13:14.154766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.174509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1116/2220] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:13:14.184092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08ae554-017e-4d53-945d-ed0fddf46d7b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.203623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.205677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.220339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.220339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.220339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.220339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.258615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:14.338272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1105/2221] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:13:14 2022] POST /identity/v3/projects => generated 373 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:13:14.344432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.376147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee043c46-5b0c-4c2c-8672-bb5a5c01d480 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.524436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.525322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.525606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.525890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.526572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.528951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.559096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.562712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:14.581764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1117/2222] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:13:14 2022] POST /identity/v3/users => generated 287 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:13:14.587492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.597385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0732f3-ff94-4008-8ca4-4a02ec62622f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.609677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1073560-20fb-4a7d-bdd4-e0cb7a9e3894 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.609907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.612085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.613092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.613805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] PATH_INFO: `/v3/projects/4203fc81bbd14439bb6ce329a5ccbc79` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.618540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4203fc81bbd14439bb6ce329a5ccbc79)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.624111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.624111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.624111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.624111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.625432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.625432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.629239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.634272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:14.648336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1118/2223] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:14 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:14.666574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.685887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75636a68-c60e-4d32-81d6-3233dc79724a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.699552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:14.700753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:14.701639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:14.702185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:14.703080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:14.705913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:14.742363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1119/2224] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:14 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:14.743443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:14.763136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4203fc81bbd14439bb6ce329a5ccbc79', 'request_id': 'req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:14.764623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc4ee69-198c-48e1-a6d9-bc787b38049b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:14.797538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4203fc81bbd14439bb6ce329a5ccbc79', 'request_id': 'req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:14.822894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4203fc81bbd14439bb6ce329a5ccbc79', 'request_id': 'req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:14.868316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] The token cache is being invalidate because project 4203fc81bbd14439bb6ce329a5ccbc79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:13:14.868316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:14.873841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c0357e60-7b0a-4bdd-85e6-a61480dfb0b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:14.876479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1106/2225] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:13:14 2022] DELETE /identity/v3/projects/4203fc81bbd14439bb6ce329a5ccbc79 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:15.025107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:15.070914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:15.074307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:15.074580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] PATH_INFO: `/v3/projects/9188a7a37dd8406cb09209ca2db405a0/users/22adaeab39234bab8ea57bdb55f70ec2/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:15.165872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9188a7a37dd8406cb09209ca2db405a0, user_id=22adaeab39234bab8ea57bdb55f70ec2, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:15.184594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:15.224049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1120/2226] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:13:14 2022] PUT /identity/v3/projects/9188a7a37dd8406cb09209ca2db405a0/users/22adaeab39234bab8ea57bdb55f70ec2/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:15.241156 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:15.241836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:15.242270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:15.294976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] MFA Rules not processed for user `22adaeab39234bab8ea57bdb55f70ec2`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:15.296827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:15.399480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1107/2227] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:15.417187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:15.419383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:15.521515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f148770-18a3-448c-a474-6f7e2cbfe271 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:15.528302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b2cab5-d625-4583-bd62-22a293d2dd09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:15.529480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b2cab5-d625-4583-bd62-22a293d2dd09 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:15.530027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b2cab5-d625-4583-bd62-22a293d2dd09 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:15.530400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b2cab5-d625-4583-bd62-22a293d2dd09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:15.549566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b2cab5-d625-4583-bd62-22a293d2dd09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:15.551438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b2cab5-d625-4583-bd62-22a293d2dd09 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:15.564317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1121/2228] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:15.607757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:15.610565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:15.729596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cd6ca4-17dd-41ab-b2c6-caa139de795f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:15.739163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:15.740020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:15.740331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:15.740690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] PATH_INFO: `/v3/users/22adaeab39234bab8ea57bdb55f70ec2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:15.742206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=22adaeab39234bab8ea57bdb55f70ec2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:15.744614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:15.753155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:15.831864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22adaeab39234bab8ea57bdb55f70ec2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:15.843452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22adaeab39234bab8ea57bdb55f70ec2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:15.859581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22adaeab39234bab8ea57bdb55f70ec2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:15.860876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22adaeab39234bab8ea57bdb55f70ec2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:15.863838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22adaeab39234bab8ea57bdb55f70ec2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:15.917708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1108/2229] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:13:15 2022] DELETE /identity/v3/users/22adaeab39234bab8ea57bdb55f70ec2 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:15.927980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:15.928540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:15.929049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:15.962972 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:15.965164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:16.059023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1122/2230] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:16.073727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:16.075579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:16.185171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051ac7ce-a3a7-445b-a61f-98e0d5e44c9d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:16.191772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e43921-13a0-4152-a44d-3254e0462449 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:16.192523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e43921-13a0-4152-a44d-3254e0462449 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:16.192805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e43921-13a0-4152-a44d-3254e0462449 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:16.193083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e43921-13a0-4152-a44d-3254e0462449 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:16.204732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e43921-13a0-4152-a44d-3254e0462449 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:16.206280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e43921-13a0-4152-a44d-3254e0462449 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:16.221629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1109/2231] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:13:17.169616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:17.172069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:17.222141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b13cd-7366-45f0-893d-ce563b0d9994 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:17.229864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:17.230969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:17.231452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:17.231948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] PATH_INFO: `/v3/projects/9188a7a37dd8406cb09209ca2db405a0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:17.234355 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9188a7a37dd8406cb09209ca2db405a0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:17.238220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:17.294837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:17.420506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9188a7a37dd8406cb09209ca2db405a0', 'request_id': 'req-c6c7b936-a8aa-4567-a59a-893dee333842'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:17.429433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9188a7a37dd8406cb09209ca2db405a0', 'request_id': 'req-c6c7b936-a8aa-4567-a59a-893dee333842'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:17.444747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9188a7a37dd8406cb09209ca2db405a0', 'request_id': 'req-c6c7b936-a8aa-4567-a59a-893dee333842'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:17.482803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] The token cache is being invalidate because project 9188a7a37dd8406cb09209ca2db405a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:13:17.483506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:17.484805 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6c7b936-a8aa-4567-a59a-893dee333842 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:17.492703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1123/2232] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:13:17 2022] DELETE /identity/v3/projects/9188a7a37dd8406cb09209ca2db405a0 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:17.495016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:17.495944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:17.495944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:17.542163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:17.545559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:17.638939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1110/2233] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:17.652960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:17.653678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:17.654136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:17.689379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:17.692636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:17.710921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1124/2234] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:17.715189 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:17.737530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c05c14e-6be0-4650-aaf0-cab66a5840f3 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:17.749422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:17.750426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:17.750620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:17.750957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:17.751602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:13:17.752046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:17.754094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:17.787116 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:17.788623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1111/2235] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:13:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:13:17.813793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b08137-86b8-43e8-a9d8-4b38ad8f6f44 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:17.823034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:17.824671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:17.825242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:17.825885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:17.827379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:17.832214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:17.918122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1125/2236] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:13:17 2022] POST /identity/v3/projects => generated 383 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:13:17.924682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:17.946380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f2c97e-3b01-4e13-bc14-b7ec6118b291 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:17.966810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:18.143352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:18.144365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:18.144872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:18.145300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:18.146134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:18.148398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:18.206758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1112/2237] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:13:17 2022] POST /identity/v3/users => generated 292 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:13:18.213674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:18.255329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2819601-3e2f-4ee7-be26-771e5acb34cc admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:18.293906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:18.295056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:18.295551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:18.296031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:18.297377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:18.322265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:18.346078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1126/2238] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:18.352303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:18.380738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfaf838-9846-453e-acf0-63ea3505b607 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:18.404120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148cf5c4-2b4f-4180-8e2b-e28482c83785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:18.417369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148cf5c4-2b4f-4180-8e2b-e28482c83785 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:18.418149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148cf5c4-2b4f-4180-8e2b-e28482c83785 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:18.419023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148cf5c4-2b4f-4180-8e2b-e28482c83785 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:18.421948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148cf5c4-2b4f-4180-8e2b-e28482c83785 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:18.427182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148cf5c4-2b4f-4180-8e2b-e28482c83785 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:18.476913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:18.477688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1113/2239] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:18.500625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7603de2-d60b-45ff-be50-d5860bb705d4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:18.508605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:18.510089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:18.510692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:18.519085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] PATH_INFO: `/v3/projects/f7b08cc6d4a34961bdf464b58946a02e/users/6dd0ed2144164190b380ed58b3dd7cd4/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:18.553300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7b08cc6d4a34961bdf464b58946a02e, user_id=6dd0ed2144164190b380ed58b3dd7cd4, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:18.556127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:18.573724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1127/2240] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:13:18 2022] PUT /identity/v3/projects/f7b08cc6d4a34961bdf464b58946a02e/users/6dd0ed2144164190b380ed58b3dd7cd4/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:18.580981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:18.582972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:18.583706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:18.620577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:18.623109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:18.745011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1114/2241] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:18.761166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:18.766526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:18.874836 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7f8a62-aa7a-476b-a9fa-5224ac859a9d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:18.883700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2556ab-cbf8-4cc7-a5f5-48a36275fe1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:18.885019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2556ab-cbf8-4cc7-a5f5-48a36275fe1b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:18.885516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2556ab-cbf8-4cc7-a5f5-48a36275fe1b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:18.886197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2556ab-cbf8-4cc7-a5f5-48a36275fe1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:18.903874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2556ab-cbf8-4cc7-a5f5-48a36275fe1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:18.905946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2556ab-cbf8-4cc7-a5f5-48a36275fe1b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:18.927801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1128/2242] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:20.624268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:20.624268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:20.664370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:20.665417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:20.665417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:20.669009 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839ef0e2-6246-4d19-ae07-245824878787 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:20.680480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:20.682192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:20.683002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:20.683724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] PATH_INFO: `/v3/users/7995c2d62dee44d2bf35dc22a08d1afb` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:20.686255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=7995c2d62dee44d2bf35dc22a08d1afb)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:20.692292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:20.763075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] MFA Rules not processed for user `6dd0ed2144164190b380ed58b3dd7cd4`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:20.765883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:20.847204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:20.897518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7995c2d62dee44d2bf35dc22a08d1afb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:20.914148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7995c2d62dee44d2bf35dc22a08d1afb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:20.938179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7995c2d62dee44d2bf35dc22a08d1afb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:20.939449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7995c2d62dee44d2bf35dc22a08d1afb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:20.940783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a07bdb0-66a0-4bae-b410-63171c4703ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7995c2d62dee44d2bf35dc22a08d1afb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:20.972104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1129/2243] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:20.985904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:20.989310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:21.004412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1115/2244] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:13:20 2022] DELETE /identity/v3/users/7995c2d62dee44d2bf35dc22a08d1afb => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:21.008683 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:21.009135 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:21.009621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:21.049943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:21.091575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:21.094248 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:21.202440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f4659c-7f7c-4a69-ab7f-c8a85bbf2632 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:21.225024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1116/2245] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:21.237720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:21.239327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:21.420002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:21.420594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:21.421120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:21.421611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:21.427251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d7da67-116b-4b53-81f8-ec64eb5056dc None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:21.440629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:21.454666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:21.456573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:21.457557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:21.458359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:21.483754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:21.486995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:21.544891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1117/2246] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:21.692733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:21.704480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e0998e-8f76-4506-b9f0-75d9675e71da service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:21.739065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1130/2247] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 754 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:13:22.491555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:22.495031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:22.536764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3d4753-508f-4d56-9730-df8609ad8601 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:22.547328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:22.548389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:22.549693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:22.550086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] PATH_INFO: `/v3/projects/c904b341e34c4b64a2b4eb8772d19279` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:22.553180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c904b341e34c4b64a2b4eb8772d19279)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:22.573235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:22.777370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c904b341e34c4b64a2b4eb8772d19279', 'request_id': 'req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:22.798905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c904b341e34c4b64a2b4eb8772d19279', 'request_id': 'req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:22.818848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c904b341e34c4b64a2b4eb8772d19279', 'request_id': 'req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:22.848509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] The token cache is being invalidate because project c904b341e34c4b64a2b4eb8772d19279 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:13:22.848995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:22.850666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3031ba0d-076a-46c0-8df8-b6ea25e1c66c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:13:22.853646 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1118/2248] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:13:22 2022] DELETE /identity/v3/projects/c904b341e34c4b64a2b4eb8772d19279 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:22.858556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:22.859515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:22.859989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:22.894447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:22.896962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:22.973379 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:23.016058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1131/2249] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:23.027118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.027568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.027866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.053889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:23.055462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:23.065878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1119/2250] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:13:23.080722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.096020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83429f7d-40cb-4da9-92d5-4042d87f1541 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.105127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.106192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.106742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.107160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.108440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:13:23.108981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.112125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.135739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1132/2251] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:13:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:13:23.140799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.153620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26518e0-c778-4ea1-b328-0650a1b79a07 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.160668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.161435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.161793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.162167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.162883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.165752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.209742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1120/2252] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:13:23 2022] POST /identity/v3/projects => generated 379 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:13:23.215911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.239490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b947e6ef-27c8-4351-b3b1-05e3617ac8b2 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.363558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.364618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.365332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.366044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.367246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.370876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.425597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1133/2253] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:13:23 2022] POST /identity/v3/users => generated 290 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:13:23.432296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.443042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:23.454733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae95eaa-258f-4137-8658-2194ba537dc9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.464430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.465359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.465784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.466273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.467288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.469346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.488791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1121/2254] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:23 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:23.493118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.504755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acd8fce-8a46-4acf-94be-d1a1784503c8 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.510516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb77803-5411-4713-b555-fdc3dbae0828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.512356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb77803-5411-4713-b555-fdc3dbae0828 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.512683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb77803-5411-4713-b555-fdc3dbae0828 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.512982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb77803-5411-4713-b555-fdc3dbae0828 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.513660 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb77803-5411-4713-b555-fdc3dbae0828 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.515625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb77803-5411-4713-b555-fdc3dbae0828 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.531722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1134/2255] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:23 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:23.539743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.548497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f0784-9b5f-4c70-bec1-8bbe002d5af5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.553626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.554421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.554817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.555268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] PATH_INFO: `/v3/projects/b73b98c606a34d1faf0a31a9f6c59e56/users/642126380bcf4446bf1e320f93139034/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.581960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b73b98c606a34d1faf0a31a9f6c59e56, user_id=642126380bcf4446bf1e320f93139034, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.585625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.603482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1122/2256] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:13:23 2022] PUT /identity/v3/projects/b73b98c606a34d1faf0a31a9f6c59e56/users/642126380bcf4446bf1e320f93139034/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:23.609161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5326708d-4be0-452b-841e-10284b18d302 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.609515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5326708d-4be0-452b-841e-10284b18d302 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.609823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5326708d-4be0-452b-841e-10284b18d302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.634938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5326708d-4be0-452b-841e-10284b18d302 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:23.635198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5326708d-4be0-452b-841e-10284b18d302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:23.731241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1135/2257] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:23.743783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:23.745814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:23.834494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004630a0-55b7-4835-ae2d-88fbcdcf9155 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:23.839115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0b87f2-ba60-49e0-a5fd-c967d9e76cd5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:23.839755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b87f2-ba60-49e0-a5fd-c967d9e76cd5 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.840039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b87f2-ba60-49e0-a5fd-c967d9e76cd5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.840400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0b87f2-ba60-49e0-a5fd-c967d9e76cd5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:23.853256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0b87f2-ba60-49e0-a5fd-c967d9e76cd5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:23.854866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0b87f2-ba60-49e0-a5fd-c967d9e76cd5 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:23.877103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1123/2258] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:23.972106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:23.972753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:23.973122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:24.007962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] MFA Rules not processed for user `6dd0ed2144164190b380ed58b3dd7cd4`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:24.010294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:24.094732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:24.134349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1136/2259] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:33.862310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=40,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:34.121767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:34.260438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:34.261393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:34.261791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:34.348317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] MFA Rules not processed for user `642126380bcf4446bf1e320f93139034`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:34.351595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:34.576691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1124/2260] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:34.583332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:34.631167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f56d333-5538-46dd-90b3-2be81f1e59c1 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:34.644006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:34.645620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:34.646471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:34.647242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:34.663047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:34.681302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:34.885690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1137/2261] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:13:34 2022] POST /identity/v3/projects => generated 379 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:13:34.890833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:34.930738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e58f95-949e-41d2-b111-c7e1ca8eb638 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:35.119139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:35.120301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:35.121814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:35.136768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:35.138924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:35.143401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:35.219918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1125/2262] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:13:34 2022] POST /identity/v3/users => generated 296 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:13:35.225104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:35.253040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9be786-dc35-44cd-a458-82247ec496db admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:35.277409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:35.278546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:35.289297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:35.290132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:35.291409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:35.294769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:35.316228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1138/2263] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:35 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:35.321158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:35.336421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836e439c-2693-4749-bb88-f699fd05ea69 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:35.343033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:35.344722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:35.345357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:35.346529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] PATH_INFO: `/v3/projects/fa7a788fc55e4ab1ab827f71901560cf/users/75350c49b8264c35badff3288257779c/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:35.371628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7a788fc55e4ab1ab827f71901560cf, user_id=75350c49b8264c35badff3288257779c, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:35.375582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:35.398360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1126/2264] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:13:35 2022] PUT /identity/v3/projects/fa7a788fc55e4ab1ab827f71901560cf/users/75350c49b8264c35badff3288257779c/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:35.402277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:35.424473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b44bbaa-b434-4922-a7ed-44f71bb93a13 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:35.509065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553edfe5-05c6-4ebd-a3ac-682b782ce37c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:35.509719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553edfe5-05c6-4ebd-a3ac-682b782ce37c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:35.510004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553edfe5-05c6-4ebd-a3ac-682b782ce37c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:35.512085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553edfe5-05c6-4ebd-a3ac-682b782ce37c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:35.512502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553edfe5-05c6-4ebd-a3ac-682b782ce37c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:35.514470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553edfe5-05c6-4ebd-a3ac-682b782ce37c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:35.531949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1139/2265] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:13:35 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:13:35.538126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:35.559677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6a6ff6-a11a-4a74-81c0-1ecf4580c3b1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:35.569186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:35.570145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:35.570620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:35.571304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] PATH_INFO: `/v3/projects/fa7a788fc55e4ab1ab827f71901560cf/users/75350c49b8264c35badff3288257779c/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:35.574404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7a788fc55e4ab1ab827f71901560cf, user_id=75350c49b8264c35badff3288257779c, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:35.577600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:35.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1127/2266] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:13:35 2022] PUT /identity/v3/projects/fa7a788fc55e4ab1ab827f71901560cf/users/75350c49b8264c35badff3288257779c/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:13:38.386344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88647e8b-5f64-4a97-aa66-44f41ff7f832 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:38.387139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88647e8b-5f64-4a97-aa66-44f41ff7f832 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:38.388132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88647e8b-5f64-4a97-aa66-44f41ff7f832 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:38.427953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-88647e8b-5f64-4a97-aa66-44f41ff7f832 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:38.430565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-88647e8b-5f64-4a97-aa66-44f41ff7f832 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:38.562776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1140/2267] 10.222.0.26 () {58 vars in 967 bytes} [Sun Aug 7 01:13:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:38.601059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:38.605018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:38.741234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:38.822293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202ce454-250f-4663-babe-a4dd154a2f67 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:38.829282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:38.830903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:38.831403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:38.832319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:38.844955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:38.848414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:38.880534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1128/2268] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:13:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:13:43.651319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90fccb-1221-4225-8f34-b85a9e4feee0 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:43.652120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90fccb-1221-4225-8f34-b85a9e4feee0 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:43.652312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90fccb-1221-4225-8f34-b85a9e4feee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:43.685573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9c90fccb-1221-4225-8f34-b85a9e4feee0 None None] MFA Rules not processed for user `75350c49b8264c35badff3288257779c`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:13:43.686839 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9c90fccb-1221-4225-8f34-b85a9e4feee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:43.828476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:43.852904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1141/2269] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:13:43 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:13:43.871877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:13:43.874045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:13:44.001387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184c5689-f513-412a-96f9-3375593018c6 service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:13:44.009128 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c21a64-6a58-4527-9716-75568967a36c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:13:44.009771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c21a64-6a58-4527-9716-75568967a36c service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:13:44.010057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c21a64-6a58-4527-9716-75568967a36c service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:13:44.010338 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c21a64-6a58-4527-9716-75568967a36c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:13:44.115678 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c21a64-6a58-4527-9716-75568967a36c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:13:44.117568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c21a64-6a58-4527-9716-75568967a36c service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:13:44.129779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1129/2270] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:13:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:13:53.849471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:13:54.124274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=69 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:14:01.592830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2de982d-3218-4eab-a412-41613b6ed8dc None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:01.599078 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2de982d-3218-4eab-a412-41613b6ed8dc None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:01.600334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2de982d-3218-4eab-a412-41613b6ed8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:01.686477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-c2de982d-3218-4eab-a412-41613b6ed8dc None None] MFA Rules not processed for user `ec931c9cc56a462991413926d34dfa46`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:14:01.692552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c2de982d-3218-4eab-a412-41613b6ed8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:01.840021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1142/2271] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:14:01 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:14:11.822608 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=25 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:14:15.560901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:15.562782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:15.563227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:15.589316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] MFA Rules not processed for user `e65f0f1ba54f483abff788974b1256cb`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:14:15.591354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:15.701272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1130/2272] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:14:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:14:16.207943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cce56-5841-41d9-aed9-a34fd5127aa4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:16.208548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cce56-5841-41d9-aed9-a34fd5127aa4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:16.208983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cce56-5841-41d9-aed9-a34fd5127aa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:16.236907 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-de3cce56-5841-41d9-aed9-a34fd5127aa4 None None] MFA Rules not processed for user `642126380bcf4446bf1e320f93139034`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:14:16.238138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-de3cce56-5841-41d9-aed9-a34fd5127aa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:16.246051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1143/2273] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:14:16 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:14:25.692814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=25 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:14:26.230504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:14:46.997518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:14:47.000545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:47.137458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef48f74-73ec-4733-983d-58396c965f63 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:14:47.148437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:14:47.149941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:47.150597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:47.151093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:47.170763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:14:47.173559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:14:47.187411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1131/2274] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:14:46 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:14:51.968181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:51.969485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:51.969963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:51.997977 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] MFA Rules not processed for user `5fafe42957914b929f9ffee53ab6e7ac`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:14:51.999372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:52.096586 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1144/2275] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:14:51 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:14:56.547477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:14:56.557341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45cd9d8-7229-44de-a98a-99c3dfce55c0 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:14:56.561461 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:14:56.562122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:56.562433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:56.562735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:56.565214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:56.580700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:14:56.582148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddb2137-8f17-4c7d-a34a-27dccc44b0a9 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:14:56.596357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1132/2276] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:14:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:14:57.903892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:14:57.921368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd97fb0-4e68-4853-af43-c4706a8dbc9b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:14:57.930401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:14:57.931445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:57.931933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:57.932450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:57.935616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:57.954008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:14:57.955564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:14:57.968965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1145/2277] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:14:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:14:59.536577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3037bf0f-37fb-45c3-b1f3-6fed4fa13313 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:14:59.537205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3037bf0f-37fb-45c3-b1f3-6fed4fa13313 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:14:59.537205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3037bf0f-37fb-45c3-b1f3-6fed4fa13313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:14:59.560845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3037bf0f-37fb-45c3-b1f3-6fed4fa13313 None None] MFA Rules not processed for user `6dd0ed2144164190b380ed58b3dd7cd4`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:14:59.562221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3037bf0f-37fb-45c3-b1f3-6fed4fa13313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:14:59.644968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1133/2278] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:14:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:15:05.969645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:15:05.984712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c8f4cf-9e26-4448-b154-ce2d2f2df39b service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:15:05.993150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:15:05.994791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:15:05.995702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:15:05.996345 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:15:06.000037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:15:06.022651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:15:06.024752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:15:06.045251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1146/2279] 10.222.0.26 () {60 vars in 1396 bytes} [Sun Aug 7 01:15:05 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3264 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:15:09.637049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=57 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:15:13.210668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:15:13.212847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:15:13.213674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:15:13.244093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] MFA Rules not processed for user `4ae9c7ef17b746fc8f87e3c22a241b76`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:15:13.246557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:15:13.358531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1134/2280] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:15:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:15:16.035549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=37 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:15:21.963688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:15:21.983607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadaf6f9-2896-44a8-951b-8af35eab858e service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:15:21.992202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:15:21.993143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:15:21.993536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:15:21.993873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:15:21.995920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:15:22.010153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:15:22.012287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:15:22.026979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1147/2281] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:15:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:15:23.348548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=25 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:15:32.019047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:15:32.601813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:15:32.623161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3780a-41ec-41ed-8655-3a919ff5351e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:15:32.632853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:15:32.633894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:15:32.634374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:15:32.634828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:15:32.637909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:15:32.658613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:15:32.660994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bbfc6b-f481-4f78-aaa8-b5f8e0144d40 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:15:32.680060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1135/2282] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:15:32 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:15:42.671728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:16:18.497592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:16:18.516829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b46b748-d733-4470-bc5f-54cc1637a34b service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:16:18.523797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:16:18.524790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:16:18.525568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:16:18.526051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:16:18.529607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:16:18.548101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:16:18.550956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:16:18.568173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1148/2283] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:16:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:16:28.560081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:16:36.147239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:16:36.148118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:16:36.148304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:16:36.183749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] MFA Rules not processed for user `ec931c9cc56a462991413926d34dfa46`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:16:36.185801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:16:36.198685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1136/2284] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:16:36 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:16:42.969197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:16:43.011774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd9887-6677-48be-af48-fe52cb42353b service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:16:43.025562 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:16:43.027186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:16:43.028014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:16:43.028841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:16:43.033184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:16:43.064106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:16:43.068626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:16:43.090995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1149/2285] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:16:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:16:46.175841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:16:53.085548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:17:02.464054 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:17:02.484541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01616a0a-7c39-40a1-a48c-8327fe536a59 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:17:02.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:17:02.496037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:17:02.496712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:17:02.497055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:17:02.499559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:17:02.519652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:17:02.522205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:17:02.547913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1137/2286] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:17:02 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:17:12.531350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:17:15.214457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:17:15.216077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:17:15.282533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7f9322-38fd-44f0-8ff5-dc4a706ca3e9 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:17:15.286722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:17:15.287384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:17:15.287748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:17:15.288029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:17:15.290048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:17:15.303687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:17:15.305333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:17:15.315996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1150/2287] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:17:15 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:17:25.311469 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=26 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:18:43.887903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:18:44.012067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f4d42-3c50-4733-af5e-6fef126e3fcc service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:18:44.021329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:18:44.022345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:18:44.022818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:18:44.023305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:18:44.038477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:18:44.041613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:18:44.056521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1138/2288] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:18:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:18:54.050928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=30 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:18:59.263556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:18:59.265930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:18:59.446494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d19513-4284-40f8-b790-6bec7cfa6410 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:18:59.453238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:18:59.453938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:18:59.454249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:18:59.454564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:18:59.456637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:18:59.471689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:18:59.474016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:18:59.489274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1151/2289] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:18:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:19:02.778965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:19:02.781059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:19:02.843976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7126b704-68f6-4606-9cde-5a2ff960bcb6 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:19:02.859892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:19:02.860840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:19:02.861380 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:19:02.861959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:19:02.865133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:19:02.886049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:19:02.888902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:19:02.907394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1139/2290] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:19:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:19:09.483111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=56 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:19:12.898671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=15 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:19:56.035261 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:19:56.065986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fd6586-58bd-43de-b837-9d0fcf677d5b service placement] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:19:56.071819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:19:56.072704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:19:56.073039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:19:56.073375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:19:56.090619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:19:56.092729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:19:56.114010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1152/2291] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:19:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:20:00.735279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:20:00.747882 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd84eef-518d-4d63-b996-4ecf00e84d21 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:20:00.753818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:20:00.754583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:20:00.754959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:20:00.755325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:20:00.768930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:20:00.770688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:20:00.785106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1140/2292] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:20:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:20:06.103581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=10 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:20:10.778792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:21:03.147356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:21:03.163970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd24841-34a1-44ab-aa6d-50c6bd4c80fc service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:21:03.169221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:21:03.170179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:21:03.170668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:21:03.171149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:21:03.188749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:21:03.191566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:21:03.244674 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1153/2293] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:21:03 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:21:03.300899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:21:03.313669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa53ac-5f87-4d92-adfa-fccca83a834e service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:21:03.318701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:21:03.319325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:21:03.319602 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:21:03.319871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:21:03.336842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:21:03.339623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:21:03.393333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1141/2294] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:21:03 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:21:03.445224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:21:03.456860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62dde2-6891-4100-a5a5-a885e19b30cc service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:21:03.461134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:21:03.461781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:21:03.462139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:21:03.462444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:21:03.474193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:21:03.475727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:21:03.510414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1154/2295] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:21:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:21:13.387591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=13 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:21:13.507060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=26 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:21:58.847923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:21:58.862749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794b98cf-ba71-4740-aa00-01bf645c03ef service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:21:58.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:21:58.867977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:21:58.868280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:21:58.868553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:21:58.877482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:21:58.878997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:21:58.889117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1142/2296] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:21:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:22:08.884777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:22:15.151036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:22:15.153424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:22:15.212610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410a9d1c-6f72-4b46-a78b-9b2cc74ffd97 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:22:15.223518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:22:15.224636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:22:15.225172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:22:15.225665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:22:15.243536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:22:15.246754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:22:15.310115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1155/2297] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:22:15 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:22:25.303352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=18 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:23:07.493720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:23:07.518295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9c3125-61fe-4eee-9660-c4e7b3c000e8 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:23:07.529660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a33d2e-14ea-4f82-b67f-d6863c0fda43 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:23:07.531203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a33d2e-14ea-4f82-b67f-d6863c0fda43 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:07.531967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a33d2e-14ea-4f82-b67f-d6863c0fda43 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:07.532835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a33d2e-14ea-4f82-b67f-d6863c0fda43 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:07.551823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a33d2e-14ea-4f82-b67f-d6863c0fda43 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:23:07.554908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a33d2e-14ea-4f82-b67f-d6863c0fda43 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:23:07.580139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1143/2298] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:23:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:23:08.070459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:23:08.093240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4e786d-c780-4383-bc9b-a9fdd3bd63eb service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:23:08.114606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:23:08.115643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:08.116225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:08.116806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:08.134340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:23:08.137141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:23:08.155027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1156/2299] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:23:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:23:13.993200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:13.993798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:13.993798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:14.040619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] MFA Rules not processed for user `e65f0f1ba54f483abff788974b1256cb`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:23:14.041870 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:23:14.050628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1144/2300] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:23:13 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:23:14.427861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:23:14.445124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f4784-62b9-4cba-a9f4-2d4142fd2312 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:23:14.451672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:23:14.452762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:14.453246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:14.453734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:14.457033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:23:14.504393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:23:14.507333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:23:14.522799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1157/2301] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:23:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:23:14.980732 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:23:14.996613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd3addc-44b1-4949-b91a-daab9e088379 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:23:15.003551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:23:15.004505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:15.004788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:15.005066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:15.007406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:23:15.023745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:23:15.025822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:23:15.040499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1145/2302] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:23:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:23:24.278285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:23:24.300769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80c6862-3567-4286-af84-3566b335c59b service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:23:24.308288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:23:24.309633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:24.310163 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:24.310594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:24.314309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:23:24.338470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:23:24.341931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:23:24.364454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1158/2303] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:23:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:23:25.033950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=23 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:23:34.355000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=23 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:23:43.376457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:23:43.389931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4d047d-5dc6-40ea-8aae-bef19c79e390 service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:23:43.396017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:23:43.396779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:23:43.397222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:23:43.397511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:23:43.399768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:23:43.455201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:23:43.458122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:23:43.495537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1146/2304] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:23:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:23:53.490114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=19 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:24:04.305219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:24:04.395151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70f9e0e-5ed0-4398-9df4-565fbaf12db0 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:24:04.401869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:24:04.402611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:24:04.402939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:24:04.403256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:24:04.413953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:24:04.415616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:24:04.449843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1159/2305] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:24:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:24:06.177015 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:24:06.190736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb685d83-26db-4d81-bede-744a7499e9a9 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:24:06.196862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:24:06.197616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:24:06.197942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:24:06.198275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:24:06.211852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:24:06.213706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:24:06.227478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1147/2306] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:24:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:24:14.446351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=41 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:24:16.221906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:25:04.285081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:25:04.303381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42041126-c4ba-4b65-a8bf-6094480f6228 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:25:04.309302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:25:04.309938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:25:04.310216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:25:04.310490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:25:04.312828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:25:04.330097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:25:04.332327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:25:04.347140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1160/2307] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:25:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:25:04.640820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:25:04.657641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04248d0a-06f8-4a70-979f-3457e810b2ae service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:25:04.664118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:25:04.665191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:25:04.665663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:25:04.666147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:25:04.669750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:25:04.698492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:25:04.701175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:25:04.725326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1148/2308] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:25:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:25:14.340671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:25:14.713623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:27:00.901565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:27:00.903312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:27:00.983563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d72ae-9283-4ee5-8b35-d6a608e592c8 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:27:00.987755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:27:00.988460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:27:00.988735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:27:00.989011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:27:00.991127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:27:01.002443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:27:01.003904 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:27:01.014356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1161/2309] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:27:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:27:05.146526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:27:05.165815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaf59e0-2a49-4d23-aea6-c67c317297c9 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:27:05.173167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:27:05.173914 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:27:05.174247 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:27:05.174693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:27:05.177111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:27:05.265653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:27:05.267387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:27:05.281078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1149/2310] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:27:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:27:05.376402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:27:05.392659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c31c9c-2aaf-4cdb-8be9-30e16311e0f3 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:27:05.397616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:27:05.398324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:27:05.398778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:27:05.399122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:27:05.401468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:27:05.501839 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:27:05.503490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:27:05.517241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1162/2311] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:27:05 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:27:05.597331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:27:05.613073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a331e6-ad3a-4594-93db-28f6ba693105 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:27:05.619207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:27:05.620392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:27:05.620969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:27:05.621473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:27:05.625223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:27:05.741470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:27:05.744108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:27:05.762332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1150/2312] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:27:05 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:27:14.838395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:27:14.855434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f454ae-61b1-481e-b155-258387b7183e service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:27:14.863194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:27:14.864188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:27:14.864738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:27:14.865240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:27:15.000336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:27:15.003574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:27:15.024874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1163/2313] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:27:14 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:27:15.754016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=61 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:27:25.015101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=87 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:28:43.765225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:28:43.767946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:28:43.876299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74f693c-1972-4b56-9cc2-e1f5dab8a35b service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:28:43.880731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:28:43.881367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:28:43.881831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:28:43.882045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:28:43.963548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:28:43.965091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:28:43.976855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1151/2314] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:28:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:28:53.971362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=48 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:29:05.083555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:29:05.222615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694c24a8-a32f-4553-bdf6-fc5fa899dad7 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:29:05.227361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:29:05.228076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:29:05.228301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:29:05.228579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:29:05.230791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:29:05.389969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:29:05.391473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:29:05.402556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1164/2315] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:29:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:29:11.189529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:29:11.193065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:29:11.381643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7babb1db-c3c0-4280-94bd-4c9a1c4395dd service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:29:11.388162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:29:11.388969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:29:11.389304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:29:11.389672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:29:11.391952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:29:11.405563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:29:11.407214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:29:11.419593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1152/2316] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:29:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:29:15.397820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=76 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:29:19.968140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:29:19.989508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:29:20.069841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52dee4b-4474-4982-92fb-59ea42c3a0b8 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:29:20.078393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:29:20.079140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:29:20.079604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:29:20.080171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:29:20.084013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:29:20.104968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:29:20.107014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:29:20.121482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1165/2317] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:29:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:29:21.414213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=55 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:29:30.115809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=13 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:30:05.284289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:30:05.309675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecb7594-ff46-44e1-888d-5de68a766a28 service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:30:05.316656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:30:05.317693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:30:05.318205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:30:05.318696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:30:05.337866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:30:05.340674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:30:05.362832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1153/2318] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:30:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:30:06.506998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:30:06.508732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:30:06.521370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dfa756-f6dd-40c3-80f5-bf2a27f5e21f service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:30:06.525364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:30:06.526054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:30:06.526389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:30:06.526712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:30:06.538784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:30:06.540164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:30:06.553487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1166/2319] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:30:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:30:15.353066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=10 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:30:16.547162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:32:09.863253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:32:09.875336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca39b69-aba4-4333-ad0d-c5d2bc04d5be service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:32:09.879943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:32:09.880696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:32:09.881019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:32:09.881412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:32:09.892418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:32:09.893915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:32:09.906221 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1154/2320] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:32:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:32:14.854294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:32:14.868885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fcf83a-344e-4c59-94fb-e622f3e0904b service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:32:14.876009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:32:14.877018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:32:14.877499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:32:14.877850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:32:14.880154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:32:14.908495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:32:14.910228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:32:14.923365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1167/2321] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:32:14 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:32:19.899394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:32:24.916371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=11 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:33:19.656660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:19.658483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:33:19.689593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5dba89-8c6e-45e6-b024-9c8783d2b86d service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:19.695724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:19.696885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:19.697402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:19.697904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:19.715513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:19.717896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:19.769480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1155/2322] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:33:19 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:33:19.839864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:19.856329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b049ae-db9f-45c7-ae98-faf5a0a3d47a service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:19.863710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:19.864884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:19.865407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:19.865870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:19.882535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:19.885367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:19.940817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1168/2323] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:33:19 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:33:19.994284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:20.005270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06136c93-ed29-4857-bfad-10965f6d653f service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:20.009589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:20.010222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:20.010700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:20.011125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:20.021776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:20.023532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:20.064843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1156/2324] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:33:19 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:33:22.093220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:22.116202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f83a4a-baea-44ff-a2e3-7963182355ed service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:22.126974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:22.128329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:22.129100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:22.129854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:22.134853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:33:22.375052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:22.378114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:22.418959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1169/2325] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:33:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:33:28.069188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:28.099442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a9a047-aaf5-4d97-8475-5dbe66f57013 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:28.111702 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:28.112911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:28.113490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:28.114336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:28.117993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:33:28.156152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:28.158114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:28.178817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1157/2326] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:33:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:33:32.393635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:32.397727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:33:32.411603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ef656f-d5f4-47b1-9fbd-eb457aa4da89 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:32.419621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:32.421210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:32.421638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:32.422094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] PATH_INFO: `/v3/users/642126380bcf4446bf1e320f93139034` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:32.424215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] RBAC: Authorizing `identity:delete_user(user_id=642126380bcf4446bf1e320f93139034)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:32.427739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:32.495564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '642126380bcf4446bf1e320f93139034'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:32.496450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '642126380bcf4446bf1e320f93139034'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:32.497405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '642126380bcf4446bf1e320f93139034'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:32.508501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '642126380bcf4446bf1e320f93139034'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:32.522312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '642126380bcf4446bf1e320f93139034'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:32.554462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1170/2327] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:33:32 2022] DELETE /identity/v3/users/642126380bcf4446bf1e320f93139034 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:33:32.869856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:32.902232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358a78dd-bc07-45f6-b3bd-75dab0d9476e service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:33.004738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:33.005830 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:33.006307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:33.006904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] PATH_INFO: `/v3/projects/b73b98c606a34d1faf0a31a9f6c59e56` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:33.009049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] RBAC: Authorizing `identity:delete_project(project_id=b73b98c606a34d1faf0a31a9f6c59e56)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:33.011396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:33.090437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b73b98c606a34d1faf0a31a9f6c59e56', 'request_id': 'req-10331145-da31-4037-bda3-5f335a7362de'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.100471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b73b98c606a34d1faf0a31a9f6c59e56', 'request_id': 'req-10331145-da31-4037-bda3-5f335a7362de'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.112188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b73b98c606a34d1faf0a31a9f6c59e56', 'request_id': 'req-10331145-da31-4037-bda3-5f335a7362de'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.133643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] The token cache is being invalidate because project b73b98c606a34d1faf0a31a9f6c59e56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:33:33.134153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.135278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.137893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1158/2328] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:33:32 2022] DELETE /identity/v3/projects/b73b98c606a34d1faf0a31a9f6c59e56 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:33:33.145556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:33.147478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:33:33.162583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461527dd-17e0-449c-85c2-f988a33a0e8d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:33.169168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:33:33.238861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:33.239661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:33.239987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:33.240357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] PATH_INFO: `/v3/users/75350c49b8264c35badff3288257779c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:33.259132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75350c49b8264c35badff3288257779c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:33.261759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:33.330222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75350c49b8264c35badff3288257779c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.331466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75350c49b8264c35badff3288257779c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.332414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75350c49b8264c35badff3288257779c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.344465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75350c49b8264c35badff3288257779c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.358840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75350c49b8264c35badff3288257779c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.392675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1171/2329] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:33:33 2022] DELETE /identity/v3/users/75350c49b8264c35badff3288257779c => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:33:33.641694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:33:33.643839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:33:33.665079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10331145-da31-4037-bda3-5f335a7362de admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:33:33.671659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:33:33.780877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:33:33.781829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:33:33.782122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:33:33.782404 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] PATH_INFO: `/v3/projects/fa7a788fc55e4ab1ab827f71901560cf` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:33:33.797908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa7a788fc55e4ab1ab827f71901560cf)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:33:33.800071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:33:33.872738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa7a788fc55e4ab1ab827f71901560cf', 'request_id': 'req-64885615-f409-4057-9408-7abf7a7fd18d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.880414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa7a788fc55e4ab1ab827f71901560cf', 'request_id': 'req-64885615-f409-4057-9408-7abf7a7fd18d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.890861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa7a788fc55e4ab1ab827f71901560cf', 'request_id': 'req-64885615-f409-4057-9408-7abf7a7fd18d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.919049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] The token cache is being invalidate because project fa7a788fc55e4ab1ab827f71901560cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:33:33.919393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.920960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:33:33.923063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1159/2330] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:33:33 2022] DELETE /identity/v3/projects/fa7a788fc55e4ab1ab827f71901560cf => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:33:43.384891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=54,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:33:43.908878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:34:05.808856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:34:05.811007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:34:05.915065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a28cba7-05e6-43f6-917a-69d2a0dc0925 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:34:06.003585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:34:06.004621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:34:06.005088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:34:06.005556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:34:06.008457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:34:06.105029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:34:06.106773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:34:06.129520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1172/2331] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:34:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:34:16.126351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:34:20.175318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:34:20.177351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:34:20.263851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64885615-f409-4057-9408-7abf7a7fd18d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:34:20.269182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:34:20.269872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:34:20.270255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:34:20.270623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:34:20.272542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:34:20.287435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:34:20.289549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:34:20.302170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1160/2332] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:34:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:34:30.296230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=32 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:05.756348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:05.756348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:05.854332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedeaf1e-8741-4697-b497-3d955ecc5460 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:05.858961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:05.859600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:05.859888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:05.860161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:05.863749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:05.877323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:05.878887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:05.891399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1173/2333] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:35:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:35:06.863703 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:06.865430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:06.877821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58b96f7-eb8c-4f2c-bd67-3221eb9c8fc5 service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:06.881814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:06.882436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:06.882722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:06.882997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:06.884883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:06.900558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:06.902146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:06.915982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1161/2334] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:35:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:35:15.885829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=31 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:16.910596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:20.747607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:20.759366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:20.760692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1049f3f-45ad-4c6c-840c-8ca410d5b3a4 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:20.767373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:20.768464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:20.768879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:20.769221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:20.772130 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:20.774463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0a511-568d-434e-9299-9f8c72ca8bde service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:20.780431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:20.781401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:20.781744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:20.782080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:20.784580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:20.869805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:20.871682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:20.875931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:20.877497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:20.884893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1162/2335] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:35:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:35:20.890647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1174/2336] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:35:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:35:28.351511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:28.353955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:28.379462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e72e92-e424-48f8-a8cf-c8ebb1eacb64 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:28.387224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:28.388133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:28.388623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:28.389076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] PATH_INFO: `/v3/users/4ae9c7ef17b746fc8f87e3c22a241b76` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:28.390806 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ae9c7ef17b746fc8f87e3c22a241b76)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:28.396511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:28.496363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ae9c7ef17b746fc8f87e3c22a241b76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.506379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ae9c7ef17b746fc8f87e3c22a241b76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.517534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ae9c7ef17b746fc8f87e3c22a241b76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.518311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ae9c7ef17b746fc8f87e3c22a241b76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.518935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ae9c7ef17b746fc8f87e3c22a241b76'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.534179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:28.543466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:28.545877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1163/2337] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:35:28 2022] DELETE /identity/v3/users/4ae9c7ef17b746fc8f87e3c22a241b76 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:35:28.568142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478580be-9fde-4c8f-a642-03521f448a58 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:28.682747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:28.683544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:28.683899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:28.684281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] PATH_INFO: `/v3/users/5fafe42957914b929f9ffee53ab6e7ac` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:28.685804 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fafe42957914b929f9ffee53ab6e7ac)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:28.688458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:28.762185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fafe42957914b929f9ffee53ab6e7ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.762909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fafe42957914b929f9ffee53ab6e7ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.763688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fafe42957914b929f9ffee53ab6e7ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.770439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:28.772079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:28.782006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fafe42957914b929f9ffee53ab6e7ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.796080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2cdcaf46-9ec4-402e-97d8-ae04f62d92c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fafe42957914b929f9ffee53ab6e7ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.806097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6549911c-c553-449e-97f1-feafae90dd29 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:28.814372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:28.815275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:28.815672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:28.816042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] PATH_INFO: `/v3/projects/7fd9ab6d98a44605a57bd8ae9535a686` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:28.817520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fd9ab6d98a44605a57bd8ae9535a686)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:28.819957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:28.830528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1175/2338] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:35:28 2022] DELETE /identity/v3/users/5fafe42957914b929f9ffee53ab6e7ac => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:35:28.890457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fd9ab6d98a44605a57bd8ae9535a686', 'request_id': 'req-06600102-f713-4a67-905f-b2585e9e66e5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.898343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fd9ab6d98a44605a57bd8ae9535a686', 'request_id': 'req-06600102-f713-4a67-905f-b2585e9e66e5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.901226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:28.908937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fd9ab6d98a44605a57bd8ae9535a686', 'request_id': 'req-06600102-f713-4a67-905f-b2585e9e66e5'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.930329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] The token cache is being invalidate because project 7fd9ab6d98a44605a57bd8ae9535a686 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:35:28.930986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.932063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:28.934721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1164/2339] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:35:28 2022] DELETE /identity/v3/projects/7fd9ab6d98a44605a57bd8ae9535a686 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:35:28.943072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b653f1-9184-43ba-b4ec-1d37f90d1be8 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:28.943569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b653f1-9184-43ba-b4ec-1d37f90d1be8 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:28.944051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b653f1-9184-43ba-b4ec-1d37f90d1be8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:28.975930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:29.059649 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-50b653f1-9184-43ba-b4ec-1d37f90d1be8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:29.062147 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-50b653f1-9184-43ba-b4ec-1d37f90d1be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.101251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.103484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.181385 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1176/2340] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:35:29.192453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.192953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.193449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.217020 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06600102-f713-4a67-905f-b2585e9e66e5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.222138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.223017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.223305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.223585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] PATH_INFO: `/v3/projects/1b1d33b8c71e41e6bf23d4ad8590d2c2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.224969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b1d33b8c71e41e6bf23d4ad8590d2c2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.227193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:29.227419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.229331 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.243935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1177/2341] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:35:29.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.269966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628789c4-a190-4511-bfe0-c147403485f4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.287855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.289209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.289594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.290051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.291062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:35:29.292561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.296788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.310434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b1d33b8c71e41e6bf23d4ad8590d2c2', 'request_id': 'req-28c911f4-62bf-40bb-ad69-d156f9cc094d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:29.328892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1178/2342] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:35:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:35:29.330624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.331957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b1d33b8c71e41e6bf23d4ad8590d2c2', 'request_id': 'req-28c911f4-62bf-40bb-ad69-d156f9cc094d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:29.349065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eec9d0-3566-477b-8922-b7e3bafdd869 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.349793 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b1d33b8c71e41e6bf23d4ad8590d2c2', 'request_id': 'req-28c911f4-62bf-40bb-ad69-d156f9cc094d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:29.369745 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] The token cache is being invalidate because project 1b1d33b8c71e41e6bf23d4ad8590d2c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:35:29.370330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:29.371945 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-28c911f4-62bf-40bb-ad69-d156f9cc094d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:35:29.374747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1165/2343] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:35:29 2022] DELETE /identity/v3/projects/1b1d33b8c71e41e6bf23d4ad8590d2c2 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:35:29.384033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f09a3-2ddd-4691-90a5-a50696b806da None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.384566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f09a3-2ddd-4691-90a5-a50696b806da None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.385059 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f09a3-2ddd-4691-90a5-a50696b806da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.422044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6c5f09a3-2ddd-4691-90a5-a50696b806da None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:29.423982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6c5f09a3-2ddd-4691-90a5-a50696b806da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.469134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.469912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.470300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.470681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.471488 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.473627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.530995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1166/2344] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:35:29.538034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1179/2345] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/projects => generated 373 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:35:29.541509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.541918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.542344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.542895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.546120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.566130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:29.567692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.585974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:29.621043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:29.646025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f06e82-f1d1-44d5-9ff6-0181bbc944fd admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.651399 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.652583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.652845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.653143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.653779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.655813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.664895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1167/2346] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:35:29.671537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.683421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56189cd6-1068-46a3-8b5f-1368d8cbea3f None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.689031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1180/2347] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/users => generated 287 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:35:29.689807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.690747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.690944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.691197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.691667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.692814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:35:29.693330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.696492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.715393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1168/2348] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:35:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:35:29.721854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad631989-e02c-4cc1-a7c0-e92cfdb287c8 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.727298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.727598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.727942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.728222 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.728532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.729195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.731494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.743186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1181/2349] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:35:29 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:35:29.746307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e3a3cf-c868-457c-ad31-43141ebd9da4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.747315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.753782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.754805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.755281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.755850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.757062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.759275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec27ba5-a505-481b-bfdd-2c8c070b8212 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.760719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.765686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.766423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.766918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.767275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.768019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.769815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.783053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1182/2350] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:35:29 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:35:29.787600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.798787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99999afc-0092-4a85-bfd8-9fe9027c339f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.803650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59920bd1-c588-4bdc-bb54-c82bb3c5caad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.804360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59920bd1-c588-4bdc-bb54-c82bb3c5caad admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.804645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59920bd1-c588-4bdc-bb54-c82bb3c5caad admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.804923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59920bd1-c588-4bdc-bb54-c82bb3c5caad admin admin] PATH_INFO: `/v3/projects/32acd26b74ae470889d517d5d9b6f6ca/users/19dcb0c624f7427cb2cedffadc2f5c0f/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.809274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1169/2351] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/projects => generated 393 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:35:29.810801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:29.828720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59920bd1-c588-4bdc-bb54-c82bb3c5caad admin admin] RBAC: Authorizing `identity:create_grant(project_id=32acd26b74ae470889d517d5d9b6f6ca, user_id=19dcb0c624f7427cb2cedffadc2f5c0f, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.832035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59920bd1-c588-4bdc-bb54-c82bb3c5caad admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:29.851397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1183/2352] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:35:29 2022] PUT /identity/v3/projects/32acd26b74ae470889d517d5d9b6f6ca/users/19dcb0c624f7427cb2cedffadc2f5c0f/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:35:29.863730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.864041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.864294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.895473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc4532-ea8e-4b61-a2fd-518fd417b1c4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:29.897902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] MFA Rules not processed for user `19dcb0c624f7427cb2cedffadc2f5c0f`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:29.900115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:29.993406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:29.994288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:29.994815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:29.995166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:29.996694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:29.998804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.012784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1184/2353] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:35:30.026035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1170/2354] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:35:29 2022] POST /identity/v3/users => generated 297 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:35:30.031007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.032393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.035241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:30.044361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84808989-d4fd-4a73-a7be-d2451b705263 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.050675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.051691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.052092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.052967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.054003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.056093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.070383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1185/2355] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:35:30.074929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.081773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:30.087815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044e329-53bf-4283-a16b-e37f298c3b79 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.093591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.094220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.094501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.094778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.095445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.097146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.109934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1186/2356] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:35:30.116166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.132857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13fedd9-842f-4bea-8a2a-63db5eb2c932 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.142168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191f76f4-4d78-4aa3-ac37-b21bb22d1485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.143319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191f76f4-4d78-4aa3-ac37-b21bb22d1485 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.143821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191f76f4-4d78-4aa3-ac37-b21bb22d1485 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.144383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191f76f4-4d78-4aa3-ac37-b21bb22d1485 admin admin] PATH_INFO: `/v3/projects/8db53b739ba9452ab93916917bba9a1f/users/9ad9d6266ee24b0b9fa04bfc46cad778/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.168852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191f76f4-4d78-4aa3-ac37-b21bb22d1485 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8db53b739ba9452ab93916917bba9a1f, user_id=9ad9d6266ee24b0b9fa04bfc46cad778, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.172706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191f76f4-4d78-4aa3-ac37-b21bb22d1485 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.176063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb9f8d-d876-4ebd-95f8-a54740d034e4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.184777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.185679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.186024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.186359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.193150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1187/2357] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:35:30 2022] PUT /identity/v3/projects/8db53b739ba9452ab93916917bba9a1f/users/9ad9d6266ee24b0b9fa04bfc46cad778/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:35:30.200797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.203994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.213961 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.215025 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.215472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.222352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1171/2358] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:35:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:35:30.252057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] MFA Rules not processed for user `9ad9d6266ee24b0b9fa04bfc46cad778`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:30.253253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:30.268640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:30.337924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1188/2359] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:30 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:35:30.344767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.363531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a6a9e6-c26e-4f04-8fb2-3d6f1cbd4081 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.431295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.432188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.432460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.432680 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.433876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.435936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.470071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1172/2360] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:35:30 2022] POST /identity/v3/projects => generated 391 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:35:30.475344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.487700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121531b-79ba-4f06-a741-9748b2a27d51 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.561507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.562555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.562743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.563010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.563671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.565756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.591575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1189/2361] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:35:30 2022] POST /identity/v3/users => generated 302 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:35:30.597391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.618785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f567d-3c11-478b-aa73-fb0de70fe7e1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.626050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.627414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.627776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.628119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.628960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.631259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.647832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1173/2362] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:35:30.653532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.666872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4580745-b08d-408a-8ddf-ae33a93b90c4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.673601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.674563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.674983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.675441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] PATH_INFO: `/v3/projects/be3a83de91624ea7a0a2b1f7cda08df7/users/9a27eff104424b88aabc5d0869410f74/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.703795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be3a83de91624ea7a0a2b1f7cda08df7, user_id=9a27eff104424b88aabc5d0869410f74, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.709265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.736132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1190/2363] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:35:30 2022] PUT /identity/v3/projects/be3a83de91624ea7a0a2b1f7cda08df7/users/9a27eff104424b88aabc5d0869410f74/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:35:30.741259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.757139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76957f5-63dc-4e52-b5e4-284d7c7b5f87 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.828963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da719af-6d84-4f30-9c2b-88131ea676e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.829876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da719af-6d84-4f30-9c2b-88131ea676e3 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.830272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da719af-6d84-4f30-9c2b-88131ea676e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.830679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da719af-6d84-4f30-9c2b-88131ea676e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.831530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da719af-6d84-4f30-9c2b-88131ea676e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.833803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da719af-6d84-4f30-9c2b-88131ea676e3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.845904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1174/2364] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:35:30 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:35:30.850589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:35:30.863785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca3ec12-bfb7-4b01-8a8f-fd61ce073164 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:35:30.869248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:35:30.869935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.870219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.871036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] PATH_INFO: `/v3/projects/be3a83de91624ea7a0a2b1f7cda08df7/users/9a27eff104424b88aabc5d0869410f74/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.873271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be3a83de91624ea7a0a2b1f7cda08df7, user_id=9a27eff104424b88aabc5d0869410f74, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:35:30.876097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:35:30.889417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1191/2365] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:35:30 2022] PUT /identity/v3/projects/be3a83de91624ea7a0a2b1f7cda08df7/users/9a27eff104424b88aabc5d0869410f74/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:35:30.902877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de60ae4a-3acc-476f-aa82-5b668fcbf8f4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:35:30.903240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de60ae4a-3acc-476f-aa82-5b668fcbf8f4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:35:30.903598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de60ae4a-3acc-476f-aa82-5b668fcbf8f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:35:30.941713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-de60ae4a-3acc-476f-aa82-5b668fcbf8f4 None None] MFA Rules not processed for user `9a27eff104424b88aabc5d0869410f74`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:35:30.943785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-de60ae4a-3acc-476f-aa82-5b668fcbf8f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:35:30.953220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:31.034794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1175/2366] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:35:30 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:35:40.884500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=64,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:35:41.030993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=26 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:36:30.121126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:30.211053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864b44f-a599-44d3-82e5-3b4a9f3cc7e3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:30.216706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:30.217373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:30.217647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:30.217918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] PATH_INFO: `/v3/users/19dcb0c624f7427cb2cedffadc2f5c0f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:30.219168 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=19dcb0c624f7427cb2cedffadc2f5c0f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:30.221203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:30.278872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19dcb0c624f7427cb2cedffadc2f5c0f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.279681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19dcb0c624f7427cb2cedffadc2f5c0f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.280478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19dcb0c624f7427cb2cedffadc2f5c0f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.289998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19dcb0c624f7427cb2cedffadc2f5c0f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.299781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19dcb0c624f7427cb2cedffadc2f5c0f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.323139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1192/2367] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:36:30 2022] DELETE /identity/v3/users/19dcb0c624f7427cb2cedffadc2f5c0f => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:36:30.329690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f4807-a336-440d-becc-9772c0616d85 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:30.330288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f4807-a336-440d-becc-9772c0616d85 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:30.330288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f4807-a336-440d-becc-9772c0616d85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:30.351638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-099f4807-a336-440d-becc-9772c0616d85 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:36:30.352972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-099f4807-a336-440d-becc-9772c0616d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:30.430206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1176/2368] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:36:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:36:30.441291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:30.442714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:30.512032 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabaa1e-adc8-4075-abe8-500d708aec02 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:30.516716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d3f66a-c31e-4af9-884c-76e8aa83cac1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:30.517503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d3f66a-c31e-4af9-884c-76e8aa83cac1 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:30.517906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d3f66a-c31e-4af9-884c-76e8aa83cac1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:30.518231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d3f66a-c31e-4af9-884c-76e8aa83cac1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:30.528749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d3f66a-c31e-4af9-884c-76e8aa83cac1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:30.530288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d3f66a-c31e-4af9-884c-76e8aa83cac1 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:30.541905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1193/2369] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:36:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:36:30.865195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f4807-a336-440d-becc-9772c0616d85 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:30.867151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-099f4807-a336-440d-becc-9772c0616d85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:30.883747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f4807-a336-440d-becc-9772c0616d85 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:30.892511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:30.893590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:30.894109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:30.894591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] PATH_INFO: `/v3/projects/32acd26b74ae470889d517d5d9b6f6ca` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:30.896897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32acd26b74ae470889d517d5d9b6f6ca)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:30.900741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:30.972902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32acd26b74ae470889d517d5d9b6f6ca', 'request_id': 'req-3782a61e-1be2-497c-9938-6d8444be93d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.983257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32acd26b74ae470889d517d5d9b6f6ca', 'request_id': 'req-3782a61e-1be2-497c-9938-6d8444be93d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:30.994704 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32acd26b74ae470889d517d5d9b6f6ca', 'request_id': 'req-3782a61e-1be2-497c-9938-6d8444be93d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:31.010724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] The token cache is being invalidate because project 32acd26b74ae470889d517d5d9b6f6ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:36:31.011341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:31.012776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3782a61e-1be2-497c-9938-6d8444be93d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:36:31.015796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1177/2370] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:36:30 2022] DELETE /identity/v3/projects/32acd26b74ae470889d517d5d9b6f6ca => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:36:31.021512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.021950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.022325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.045199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:36:31.046441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:31.076317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:36:31.128803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1194/2371] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:36:31.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.141296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.141616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.168282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:36:31.169552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:31.178607 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1178/2372] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:36:31.183820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.195363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69448d6-f732-44f2-b520-e605596f5aa7 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.201623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.202260 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.202543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.202821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.203523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:36:31.203851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.205517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.221255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1195/2373] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:36:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:36:31.227114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.241656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aedc33-5023-4ec9-b9b6-896f84de6fb9 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.250658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.251701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.252222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.252751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.253905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.257710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.302804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1179/2374] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/projects => generated 373 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:36:31.310246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.321622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa69d34a-5fdf-42c9-8345-58686e2dd268 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.391293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.392188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.392665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.393080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.394140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.398057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.433267 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1196/2375] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/users => generated 287 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:36:31.437677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.454850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c133e21c-b806-4939-8815-3544bf549c5b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.462963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.464993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.465471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.465880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.466714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.468554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.481074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1180/2376] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:36:31 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:36:31.486077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.497755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf1f22-1990-4037-a05a-910b46d39392 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.502839 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7b7662-86ca-4d0f-af84-3a62c2550c07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.503465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7b7662-86ca-4d0f-af84-3a62c2550c07 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.503743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7b7662-86ca-4d0f-af84-3a62c2550c07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.504046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7b7662-86ca-4d0f-af84-3a62c2550c07 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.504745 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7b7662-86ca-4d0f-af84-3a62c2550c07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.506383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7b7662-86ca-4d0f-af84-3a62c2550c07 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.520605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1197/2377] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:36:31 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:36:31.525152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.538630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240c66e1-dc82-4ed4-9cf7-8a4b197c5232 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.545397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.546485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.546913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.547323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] PATH_INFO: `/v3/projects/59e811f0e4e54633b32d74602eef1723/users/65f4d46737694576bc36279b35b07be6/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.566161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59e811f0e4e54633b32d74602eef1723, user_id=65f4d46737694576bc36279b35b07be6, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.569100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.590550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1181/2378] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:36:31 2022] PUT /identity/v3/projects/59e811f0e4e54633b32d74602eef1723/users/65f4d46737694576bc36279b35b07be6/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:36:31.601552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.601862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.602077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.639853 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] MFA Rules not processed for user `65f4d46737694576bc36279b35b07be6`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:36:31.641787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:31.764885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1198/2379] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:36:31.769734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.794826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:36:31.847970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46525c0-b951-4ba6-b1a7-42d222a1f025 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.852986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.853613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.854419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.854696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.855360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.857342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:31.888264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1182/2380] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/projects => generated 373 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:36:31.894887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:31.907519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1d7a2-16ab-4f0d-8268-851041aa3845 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:31.923405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:36:31.987938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:31.988868 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:31.989189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:31.989463 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:31.990132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:31.992117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:32.018937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1199/2381] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:36:31 2022] POST /identity/v3/users => generated 293 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:36:32.022194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:32.033872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c8a50e-3166-48a2-ae7f-94599e6ef2f7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:32.038832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:32.039458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:32.039733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:32.040004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:32.040695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:32.042857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:32.055122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1183/2382] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:36:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:36:32.060477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:32.071242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51d2fb3-e993-4ab3-b8db-79adf0354788 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:32.076112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:32.076821 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:32.077098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:32.077370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] PATH_INFO: `/v3/projects/2aba242ce2f141868b6c2052e54ce7ff/users/504e994688904617bffb37b2ede225a0/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:32.094861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2aba242ce2f141868b6c2052e54ce7ff, user_id=504e994688904617bffb37b2ede225a0, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:32.097818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:32.115731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1200/2383] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:36:32 2022] PUT /identity/v3/projects/2aba242ce2f141868b6c2052e54ce7ff/users/504e994688904617bffb37b2ede225a0/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:36:32.122290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:32.134021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f97ce93-bbc5-4984-975f-64fe33f7880d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:32.201213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f44bf96-e4e9-41ec-9d1c-02b0ba1d382c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:32.201884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f44bf96-e4e9-41ec-9d1c-02b0ba1d382c admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:32.202204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f44bf96-e4e9-41ec-9d1c-02b0ba1d382c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:32.202512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f44bf96-e4e9-41ec-9d1c-02b0ba1d382c admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:32.203240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f44bf96-e4e9-41ec-9d1c-02b0ba1d382c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:32.205146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f44bf96-e4e9-41ec-9d1c-02b0ba1d382c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:32.217220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1184/2384] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:36:32 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:36:32.222372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:36:32.234407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53ba7fb-8b62-488c-bd5b-47224d644765 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:36:32.239510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:36:32.240207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:32.240670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:32.240954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] PATH_INFO: `/v3/projects/2aba242ce2f141868b6c2052e54ce7ff/users/504e994688904617bffb37b2ede225a0/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:32.243129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=2aba242ce2f141868b6c2052e54ce7ff, user_id=504e994688904617bffb37b2ede225a0, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:36:32.246467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:36:32.258763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1201/2385] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:36:32 2022] PUT /identity/v3/projects/2aba242ce2f141868b6c2052e54ce7ff/users/504e994688904617bffb37b2ede225a0/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:36:32.269592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:36:32.270011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:36:32.270337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:36:32.307815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] MFA Rules not processed for user `504e994688904617bffb37b2ede225a0`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:36:32.309140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:36:32.389976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1185/2386] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:36:32 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:36:42.254427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=36,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:36:42.386817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:37:14.862787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:37:14.865157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:14.948789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c03bed-5a4e-46f0-a5f6-d40a192f3f5b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:37:14.953506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:37:14.954310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:37:14.954767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:37:14.955118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:37:14.961893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:14.977737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:37:14.981092 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:37:14.998280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1202/2387] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:37:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:37:20.716928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:37:20.718583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:20.818564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6629bb32-58d3-4a49-b8ca-9543c41b4c8e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:37:20.827684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:37:20.828819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:37:20.829349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:37:20.829724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:37:20.833175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:20.989780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:37:20.991856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:37:21.017610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1186/2388] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:37:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:37:21.078196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:37:21.090200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4fb847-d0f3-47bd-bbbe-ce7ef357b538 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:37:21.094272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:37:21.095100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:37:21.095382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:37:21.095658 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:37:21.097860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:21.194216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:37:21.195702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:37:21.211276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1203/2389] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:37:21 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:37:21.349090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:37:21.350579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:21.362670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49ad8f8-2284-4e21-9331-cecad77f84b9 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:37:21.366926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:37:21.367689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:37:21.368038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:37:21.368930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:37:21.380113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:37:21.382162 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:37:21.394588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1187/2390] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:37:21 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:37:21.603735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:37:21.616062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4849e8-3cbb-485c-8005-ad910ef6bd40 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:37:21.623239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:37:21.624326 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:37:21.624817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:37:21.625302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:37:21.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:21.738532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:37:21.741546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:37:21.763790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1204/2391] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:37:21 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:37:22.358819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:37:22.364454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:22.507712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19610943-b301-4ba6-9029-5b7c595f2ca5 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:37:22.515650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:37:22.516194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:37:22.516503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:37:22.516778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:37:22.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:37:22.532839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:37:22.534471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:37:22.546744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1188/2392] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:37:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:37:31.752644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=84 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:37:32.541348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:05.620048 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:05.633715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b2da71-34d7-4096-8643-29b45ddb2d02 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:05.639393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:05.640029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:05.640358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:05.640636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:05.642507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:05.737174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:05.738836 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:05.749454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1205/2393] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:39:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:39:15.744901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=37 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:21.439749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:21.487164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2540312-04d1-474a-a8ec-9ffc3ef43177 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:21.491955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:21.492667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:21.493019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:21.493292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:21.495391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:21.576881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:21.578733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:21.620961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:21.622440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:21.635303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1189/2394] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:39:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:39:21.779475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f80efce-2182-4674-b860-f550954c23cd service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:21.799408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:21.800438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:21.800897 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:21.801343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:21.805371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:21.825539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:21.827643 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c4b87a-53d1-4f2c-b12c-3380032f815a service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:21.843397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1206/2395] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:39:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:39:22.646984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:22.718625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4eeb68-614f-434d-9bb1-7edae5ccd88b service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:22.723799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:22.724551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:22.724896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:22.725222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] PATH_INFO: `/v3/users/65f4d46737694576bc36279b35b07be6` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:22.726491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65f4d46737694576bc36279b35b07be6)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:22.728647 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:22.786266 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65f4d46737694576bc36279b35b07be6'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:22.789237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:22.796286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65f4d46737694576bc36279b35b07be6'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:22.805876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65f4d46737694576bc36279b35b07be6'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:22.806621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65f4d46737694576bc36279b35b07be6'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:22.807242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65f4d46737694576bc36279b35b07be6'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:22.832003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1190/2396] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:39:22 2022] DELETE /identity/v3/users/65f4d46737694576bc36279b35b07be6 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:39:22.838374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:22.839089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:22.839089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:22.859428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:39:22.860720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:22.940730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1207/2397] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:39:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:39:22.956319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:22.958636 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:23.060383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c188bc71-bd61-4c59-94a2-aaa364f22251 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:23.065316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:23.066104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:23.066521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:23.066952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:23.081241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:23.083106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:23.096363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1191/2398] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:39:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:39:23.999689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:24.002118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:24.027070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff31793-b49e-4c70-bfb9-b2517e2f0710 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:24.038915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:24.040265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:24.040602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:24.041088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] PATH_INFO: `/v3/projects/59e811f0e4e54633b32d74602eef1723` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:24.043956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=59e811f0e4e54633b32d74602eef1723)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:24.048397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:24.116046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:24.141702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59e811f0e4e54633b32d74602eef1723', 'request_id': 'req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.150409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59e811f0e4e54633b32d74602eef1723', 'request_id': 'req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.161781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59e811f0e4e54633b32d74602eef1723', 'request_id': 'req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.185119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] The token cache is being invalidate because project 59e811f0e4e54633b32d74602eef1723 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:39:24.185656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.186892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.189925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1208/2399] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:39:23 2022] DELETE /identity/v3/projects/59e811f0e4e54633b32d74602eef1723 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:39:24.197354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:24.215594 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72bce4d-0ace-46e3-bfc0-cb33e440ed67 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:24.298064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:24.298737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:24.299021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:24.299301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] PATH_INFO: `/v3/users/504e994688904617bffb37b2ede225a0` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:24.300653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] RBAC: Authorizing `identity:delete_user(user_id=504e994688904617bffb37b2ede225a0)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:24.302724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:24.361329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '504e994688904617bffb37b2ede225a0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.370859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '504e994688904617bffb37b2ede225a0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.381064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '504e994688904617bffb37b2ede225a0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.382616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '504e994688904617bffb37b2ede225a0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.383403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0cb0fa4b-04d7-44bd-95ce-bfacf654c293 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '504e994688904617bffb37b2ede225a0'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.392868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:24.407816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1192/2400] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:39:24 2022] DELETE /identity/v3/users/504e994688904617bffb37b2ede225a0 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:39:24.778743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:24.780595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:24.802782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a68b8-1473-4822-ada7-2fa85ee33ed3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:24.876507 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:24.877169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:24.877442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:24.877716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] PATH_INFO: `/v3/projects/2aba242ce2f141868b6c2052e54ce7ff` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:24.878899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=2aba242ce2f141868b6c2052e54ce7ff)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:24.880903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:24.943899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2aba242ce2f141868b6c2052e54ce7ff', 'request_id': 'req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.950710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2aba242ce2f141868b6c2052e54ce7ff', 'request_id': 'req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.961169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2aba242ce2f141868b6c2052e54ce7ff', 'request_id': 'req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.975194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] The token cache is being invalidate because project 2aba242ce2f141868b6c2052e54ce7ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:39:24.975528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.976851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c6f7e33f-cf05-4ff7-b66b-4d9e29cbabdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:39:24.978407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1209/2401] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:39:24 2022] DELETE /identity/v3/projects/2aba242ce2f141868b6c2052e54ce7ff => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:39:24.983838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:24.984279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:24.984658 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.006635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:39:25.007939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:25.083976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1193/2402] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:39:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:39:25.093051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.093917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.094106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.117099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:39:25.118362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:25.127579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1210/2403] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:39:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:39:25.132921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.150524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e914f32a-b781-4946-8703-f62ca7e7badc None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.158093 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.159140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.159640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.160087 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.161257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:39:25.161812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.165099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.184619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1194/2404] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:39:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:39:25.187112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.197931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99348ab-11c0-4d25-9518-45a8762fccf4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.203037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.203995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.204304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.204582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.205224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.207181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.240705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1211/2405] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:39:25 2022] POST /identity/v3/projects => generated 363 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:39:25.246353 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.260279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84580d59-7955-4821-ba4f-a057d9a8643c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.342963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.343592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.343876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.344192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.344829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.346801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.372268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1195/2406] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:39:25 2022] POST /identity/v3/users => generated 282 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:39:25.378133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.390635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45533a3a-dd06-4d79-abb1-94acd990b41c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.396034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.396735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.397009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.397302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.398091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.399797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.411330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1212/2407] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:39:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:39:25.418007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.429863 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7ec0f0-c93d-4665-974a-ae01f1d7a78f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.434795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd58ec44-c552-4ce9-a778-799f3653d435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.435479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd58ec44-c552-4ce9-a778-799f3653d435 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.435800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd58ec44-c552-4ce9-a778-799f3653d435 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.436151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd58ec44-c552-4ce9-a778-799f3653d435 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.436937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd58ec44-c552-4ce9-a778-799f3653d435 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.438687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd58ec44-c552-4ce9-a778-799f3653d435 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.452939 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1196/2408] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:39:25 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:39:25.457667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.473263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8159116-5fdf-4fb8-9bb6-08d8d34c56eb admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.482448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.484083 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.484635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] PATH_INFO: `/v3/projects/6b616599cdcd4d20a9456da6296b97c0/users/4b3fce9af3b44ff098ea41120b74259e/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.504906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:25.513780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b616599cdcd4d20a9456da6296b97c0, user_id=4b3fce9af3b44ff098ea41120b74259e, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.520445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.542284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1213/2409] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:39:25 2022] PUT /identity/v3/projects/6b616599cdcd4d20a9456da6296b97c0/users/4b3fce9af3b44ff098ea41120b74259e/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:39:25.553116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.553384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.553670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.585986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] MFA Rules not processed for user `4b3fce9af3b44ff098ea41120b74259e`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:39:25.587500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:25.651267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:25.679470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1197/2410] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:39:25 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:39:25.684437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.765734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871316d-6860-48be-8d89-8014e962494f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:25.774356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:25.775432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:25.775933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:25.776477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:25.777685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:25.781312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:25.830224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1214/2411] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:39:25 2022] POST /identity/v3/projects => generated 365 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:39:25.835632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:25.848537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf27a8-24cc-4cda-8e56-a90c15e1e1d4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.054294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.054934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.055213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.055486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.056136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.058151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.085818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1198/2412] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:39:25 2022] POST /identity/v3/users => generated 289 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:39:26.090829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:26.104017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98a23a-79bb-42ef-b08d-46010d83affd admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.111605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.112631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.113141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.113590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.114830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.118016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.138698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1215/2413] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:39:26 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:39:26.143512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:26.155716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a992475a-40c1-4f03-975c-1070a1dca59d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.162836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.164417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.164859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.165292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] PATH_INFO: `/v3/projects/3efb8ddf7d0c48698b7e9f54406414ee/users/1c15647a2b334f3bbbc10ad8cd923875/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.192970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3efb8ddf7d0c48698b7e9f54406414ee, user_id=1c15647a2b334f3bbbc10ad8cd923875, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.198385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.222972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1199/2414] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:39:26 2022] PUT /identity/v3/projects/3efb8ddf7d0c48698b7e9f54406414ee/users/1c15647a2b334f3bbbc10ad8cd923875/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:39:26.228099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:26.240938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1cbcab-587f-4fd4-ae18-78a03dc82615 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.318891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5705b53-0f7e-4d8e-8dea-dcada4cb066c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.320196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5705b53-0f7e-4d8e-8dea-dcada4cb066c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.320696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5705b53-0f7e-4d8e-8dea-dcada4cb066c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.321143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5705b53-0f7e-4d8e-8dea-dcada4cb066c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.322235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5705b53-0f7e-4d8e-8dea-dcada4cb066c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.325410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5705b53-0f7e-4d8e-8dea-dcada4cb066c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.344988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1216/2415] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:39:26 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:39:26.358273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:26.376395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1346c080-5f7d-4f31-a001-e43ca01c1396 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.382232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.383000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.383420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.383864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] PATH_INFO: `/v3/projects/3efb8ddf7d0c48698b7e9f54406414ee/users/1c15647a2b334f3bbbc10ad8cd923875/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.386165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3efb8ddf7d0c48698b7e9f54406414ee, user_id=1c15647a2b334f3bbbc10ad8cd923875, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.389254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.404649 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1200/2416] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:39:26 2022] PUT /identity/v3/projects/3efb8ddf7d0c48698b7e9f54406414ee/users/1c15647a2b334f3bbbc10ad8cd923875/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:39:26.425111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.425111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.425111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.466139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] MFA Rules not processed for user `1c15647a2b334f3bbbc10ad8cd923875`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:39:26.468272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:26.547157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:26.575105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1217/2417] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:39:26 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:39:26.588029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:26.589952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:39:26.681811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67edef33-ebeb-46db-9589-f97012411fb9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.687832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.688874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.689342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.689821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.702130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.703610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.715097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1201/2418] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:39:26 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:39:26.860182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:39:26.871404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c9ea8e-42f7-4d95-8d04-8ef56cedc582 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:39:26.875392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:39:26.876049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:39:26.876371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:39:26.876654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:39:26.943281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:39:26.945727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:39:26.957520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1218/2419] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:39:26 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:39:36.709656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=77,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:39:36.951632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=35 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:42:18.835245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:42:18.837406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:42:18.929995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9600fce-ad87-4791-9210-b8d259fef4d3 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:42:18.934166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:42:18.934790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:42:18.935099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:42:18.935379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:42:18.937329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:42:18.948002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:42:18.949513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:42:18.959795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1202/2420] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:42:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:42:20.749540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:42:20.751906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:42:20.772974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87807c1-93f4-4f84-9dfd-ddf82f246f4c service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:42:20.780015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:42:20.781237 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:42:20.781709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:42:20.782179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:42:20.785167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:42:20.870917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:42:20.872605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:42:20.884457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1219/2421] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:42:20 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:42:21.762941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:42:21.774097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dfcab4-11b5-4d58-8ed5-436d8bd06c3d service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:42:21.777919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:42:21.778685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:42:21.779005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:42:21.779391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:42:21.781697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:42:21.852418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:42:21.853875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:42:21.864561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1203/2422] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:42:21 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:42:30.878396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=33 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:42:31.859383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=58 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:23.179317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:23.193748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4005c718-052c-4d64-8a2f-d7c419f2480a service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:23.197611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:23.198233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:23.198521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:23.198786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:23.200737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:23.287807 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:23.288956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0850fb4-844d-4040-b6b2-944cd6230b1a service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:23.301664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1220/2423] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:43:23 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:43:23.494037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:23.495659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:23.658269 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb8d3b3-d607-4724-bf53-9783f5abcebe service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:23.665708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:23.666947 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:23.667136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:23.667315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] PATH_INFO: `/v3/users/9ad9d6266ee24b0b9fa04bfc46cad778` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:23.668909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ad9d6266ee24b0b9fa04bfc46cad778)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:23.675667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:23.741116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ad9d6266ee24b0b9fa04bfc46cad778'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:23.752755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ad9d6266ee24b0b9fa04bfc46cad778'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:23.762196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ad9d6266ee24b0b9fa04bfc46cad778'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:23.763027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ad9d6266ee24b0b9fa04bfc46cad778'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:23.763685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ad9d6266ee24b0b9fa04bfc46cad778'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:23.786637 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1204/2424] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:43:23 2022] DELETE /identity/v3/users/9ad9d6266ee24b0b9fa04bfc46cad778 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:43:23.792097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:23.792521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:23.792852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:23.813465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:43:23.814777 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:23.883574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1221/2425] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:43:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:43:23.894754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:23.896681 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:23.968561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccbc3a8-0108-4696-8861-849618db281e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:23.971844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:23.972730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:23.972994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:23.973265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:23.980184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:23.984894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:23.986886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:24.000146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1205/2426] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:43:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:43:24.441010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:24.443584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:24.466373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be1b7cc-b25a-4afe-a0cb-f35be574127c None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:24.475230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:24.476226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:24.476648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:24.476967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] PATH_INFO: `/v3/projects/8db53b739ba9452ab93916917bba9a1f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:24.478582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8db53b739ba9452ab93916917bba9a1f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:24.480791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:24.549465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8db53b739ba9452ab93916917bba9a1f', 'request_id': 'req-94a0795e-5fe9-4263-b2b1-72051cd79779'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.556877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8db53b739ba9452ab93916917bba9a1f', 'request_id': 'req-94a0795e-5fe9-4263-b2b1-72051cd79779'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.566706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8db53b739ba9452ab93916917bba9a1f', 'request_id': 'req-94a0795e-5fe9-4263-b2b1-72051cd79779'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.588227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] The token cache is being invalidate because project 8db53b739ba9452ab93916917bba9a1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:43:24.588882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.590052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.593264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1222/2427] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:43:24 2022] DELETE /identity/v3/projects/8db53b739ba9452ab93916917bba9a1f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:43:24.599819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:24.610904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fce9df2-e691-4a1f-bbb2-0cbd48e05961 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:24.707898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:24.708983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:24.709490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:24.709991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] PATH_INFO: `/v3/users/9a27eff104424b88aabc5d0869410f74` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:24.712487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a27eff104424b88aabc5d0869410f74)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:24.716558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:24.812834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a27eff104424b88aabc5d0869410f74'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.824354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a27eff104424b88aabc5d0869410f74'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.833033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a27eff104424b88aabc5d0869410f74'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.833941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a27eff104424b88aabc5d0869410f74'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.834671 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-1eaffb6e-6ca5-42a4-8b55-3b80f225b90e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a27eff104424b88aabc5d0869410f74'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:24.855804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1206/2428] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:43:24 2022] DELETE /identity/v3/users/9a27eff104424b88aabc5d0869410f74 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:43:25.226117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.230241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:25.296470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a0795e-5fe9-4263-b2b1-72051cd79779 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.327192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:25.403107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.404194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.404695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.405194 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] PATH_INFO: `/v3/projects/be3a83de91624ea7a0a2b1f7cda08df7` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.407190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be3a83de91624ea7a0a2b1f7cda08df7)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:25.410760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:25.483508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be3a83de91624ea7a0a2b1f7cda08df7', 'request_id': 'req-2b66b03a-08cc-4935-a426-9419180d2363'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:25.490592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be3a83de91624ea7a0a2b1f7cda08df7', 'request_id': 'req-2b66b03a-08cc-4935-a426-9419180d2363'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:25.499764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be3a83de91624ea7a0a2b1f7cda08df7', 'request_id': 'req-2b66b03a-08cc-4935-a426-9419180d2363'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:25.514571 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] The token cache is being invalidate because project be3a83de91624ea7a0a2b1f7cda08df7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:43:25.514928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:25.515664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b66b03a-08cc-4935-a426-9419180d2363 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:43:25.517646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1223/2429] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:43:25 2022] DELETE /identity/v3/projects/be3a83de91624ea7a0a2b1f7cda08df7 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:43:25.525756 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.526164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.526511 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.548229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:43:25.549892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:25.621477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1207/2430] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:43:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:43:25.632153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.632506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.632737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.655218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:43:25.656538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:25.665945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1224/2431] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:43:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:43:25.670855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.681614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca3cbdb-fc2b-464e-8fbf-d007ba6d3a5b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.686911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.687585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.687909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.688273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.689019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:43:25.689409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:25.691301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:25.702035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:25.707071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1208/2432] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:43:25.712020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.723985 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35acea4d-515a-4603-9bd3-f9060e243542 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.729225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.729915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.730270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.730588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.731284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:25.733418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:25.768360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1225/2433] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:43:25 2022] POST /identity/v3/projects => generated 405 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:43:25.773676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.785123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3509dca-3969-4a25-b61a-4b7d278f40a6 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.860231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.860890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.861170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.861442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.862095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:25.864110 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:25.888267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1209/2434] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:43:25 2022] POST /identity/v3/users => generated 303 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:43:25.893014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.904790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b92e99-a849-4078-a2cd-56c96f83d153 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.910785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.911426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.911705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.911980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.912703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:25.914365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:25.925752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1226/2435] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:43:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:43:25.930641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.941931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13d59cd-f456-49f6-bcda-6368edf086c0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.947534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfbd5c8-5a91-47c9-a0a3-df603f69e3e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.948172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfbd5c8-5a91-47c9-a0a3-df603f69e3e3 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.948493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfbd5c8-5a91-47c9-a0a3-df603f69e3e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.948799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfbd5c8-5a91-47c9-a0a3-df603f69e3e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:25.949474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfbd5c8-5a91-47c9-a0a3-df603f69e3e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:25.951186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfbd5c8-5a91-47c9-a0a3-df603f69e3e3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:25.963849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1210/2436] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:43:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:43:25.968749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:25.980386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760aa073-c944-4313-84a7-667dd50400f7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:25.985541 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:25.986276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:25.986637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:25.986962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] PATH_INFO: `/v3/projects/0886880b733e4ed8841a8ce447448728/users/842f65f130124a01b7ff8ca3f4d5cb69/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.004956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0886880b733e4ed8841a8ce447448728, user_id=842f65f130124a01b7ff8ca3f4d5cb69, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.007761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.025070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1227/2437] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:43:25 2022] PUT /identity/v3/projects/0886880b733e4ed8841a8ce447448728/users/842f65f130124a01b7ff8ca3f4d5cb69/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:43:26.036496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.036847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.037164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.068949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] MFA Rules not processed for user `842f65f130124a01b7ff8ca3f4d5cb69`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:43:26.070202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:26.155181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1211/2438] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:43:26 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:43:26.161137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.227104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:26.255987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89449f61-17e9-4202-a8c0-3fd7dfd9402f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:26.261894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:26.262689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.263255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.263452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.264117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.266384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.324014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1228/2439] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:43:26 2022] POST /identity/v3/projects => generated 407 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:43:26.330659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.350095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cacdc53-9fd5-48ad-b2e3-eaff5189e5ac None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:26.437257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:26.438219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.438668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.439126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.440199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.443768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.475652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:26.483233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1212/2440] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:43:26 2022] POST /identity/v3/users => generated 310 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:43:26.489525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.509057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce2cd38-c56a-4c01-98c5-dfc5d4e525f5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:26.516923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:26.518315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.518850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.519389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.520711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.523872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.545481 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1229/2441] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:43:26 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:43:26.551332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.569550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8a52e6-ce69-4b01-953e-306c3841ec92 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:26.577428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:26.578575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.579066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.579570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] PATH_INFO: `/v3/projects/4976d5bf279b43218ccd4cceca03b05c/users/10543d4b517b47479ac022a4f466ff10/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.607735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4976d5bf279b43218ccd4cceca03b05c, user_id=10543d4b517b47479ac022a4f466ff10, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.613737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.633409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1213/2442] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:43:26 2022] PUT /identity/v3/projects/4976d5bf279b43218ccd4cceca03b05c/users/10543d4b517b47479ac022a4f466ff10/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:43:26.638055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.649596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f617b-a7d3-4907-8c64-e79d7c17344f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:26.725749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7af092-7eaa-45d4-9921-c8030e1d2a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:26.726263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7af092-7eaa-45d4-9921-c8030e1d2a17 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.726542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7af092-7eaa-45d4-9921-c8030e1d2a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.726813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7af092-7eaa-45d4-9921-c8030e1d2a17 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.727470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7af092-7eaa-45d4-9921-c8030e1d2a17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.729149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7af092-7eaa-45d4-9921-c8030e1d2a17 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.741074 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1230/2443] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:43:26 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:43:26.747909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.766200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe6f71b-d3ab-45eb-8435-6dcac14f63e5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:26.774212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:26.775277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.775719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.776145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] PATH_INFO: `/v3/projects/4976d5bf279b43218ccd4cceca03b05c/users/10543d4b517b47479ac022a4f466ff10/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.779609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4976d5bf279b43218ccd4cceca03b05c, user_id=10543d4b517b47479ac022a4f466ff10, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:26.784787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:26.803528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1214/2444] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:43:26 2022] PUT /identity/v3/projects/4976d5bf279b43218ccd4cceca03b05c/users/10543d4b517b47479ac022a4f466ff10/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:43:26.821542 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:26.821960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:26.822348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:26.856159 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] MFA Rules not processed for user `10543d4b517b47479ac022a4f466ff10`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:43:26.857505 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:26.941015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1231/2445] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:43:26 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:43:26.958016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:26.959876 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:43:27.046309 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b16ef21-76a9-425c-b41a-a2453d634a6f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:27.051096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:27.051727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:27.052005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:27.052296 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:27.063288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:27.064852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:27.075669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1215/2446] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:43:26 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:43:27.273275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:43:27.283918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccdfe2-28df-44cc-80ae-2604cccc20f6 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:43:27.288389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:43:27.289018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:43:27.289292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:43:27.289561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:43:27.339648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:27.364442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:43:27.365963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:43:27.376824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1232/2447] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:43:27 2022] GET /identity/v3/auth/tokens => generated 3282 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:43:37.070921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=47,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:43:37.371709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=10 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:44:05.062514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:44:05.073312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4b128-fb24-4e51-9115-fec275bf6a66 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:44:05.077463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:44:05.078118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:44:05.078398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:44:05.078679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:44:05.080558 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:44:05.149824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:44:05.151305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:44:05.162426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1216/2448] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:44:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:44:15.156961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=28 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:44:20.537266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:44:20.546879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d606f30-ed86-4b2f-be89-f91974796f0b service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:44:20.551512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:44:20.552266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:44:20.552630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:44:20.552980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:44:20.555107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:44:20.638727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:44:20.640324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:44:20.663122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1233/2449] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:44:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:44:26.941568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:44:26.951542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d5e9f2-fa69-4171-8c82-bafe111c891d service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:44:26.956804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:44:26.957579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:44:26.957937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:44:26.958348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:44:26.961336 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:44:27.031045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:44:27.032765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:44:27.043615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1217/2450] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:44:26 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:44:30.659706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=36 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:44:31.180893 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:44:31.182611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:44:31.280060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51da95c-2044-404a-a518-e2f1b0466b58 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:44:31.288456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:44:31.289487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:44:31.289949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:44:31.290417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:44:31.293455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:44:31.314526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:44:31.318085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:44:31.344038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1234/2451] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:44:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:44:37.038652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=28 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:44:41.329192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=32 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:21.635147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:21.637993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:21.656016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4808c9-d8bd-4c6f-afca-35e2d7b2197e service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:21.661085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:21.661856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:21.662246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:21.662698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:21.665040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:21.801459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:21.803286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:21.822449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1218/2452] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:21 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:21.987311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:21.989383 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:22.108963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2b1a5-d519-41e8-bdf9-93b73586d25c service placement] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:22.115499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:22.116798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:22.116798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:22.116798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:22.121504 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:22.147205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:22.149713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:22.250247 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a023c-0744-4081-93ea-8c8fffb05a13 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:22.256006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:22.256718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:22.256984 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:22.257251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] PATH_INFO: `/v3/users/4b3fce9af3b44ff098ea41120b74259e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:22.259032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b3fce9af3b44ff098ea41120b74259e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:22.260668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:22.290451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:22.293255 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8fb513-3f66-4d1a-9e75-10f236d54da1 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:22.327942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b3fce9af3b44ff098ea41120b74259e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:22.336433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1235/2453] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:22.337597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b3fce9af3b44ff098ea41120b74259e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:22.348679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b3fce9af3b44ff098ea41120b74259e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:22.350058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b3fce9af3b44ff098ea41120b74259e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:22.350985 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b3fce9af3b44ff098ea41120b74259e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:22.374124 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1219/2454] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:22 2022] DELETE /identity/v3/users/4b3fce9af3b44ff098ea41120b74259e => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:22.380589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56018c-3609-4929-a270-684d3706028b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:22.381076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56018c-3609-4929-a270-684d3706028b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:22.381536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc56018c-3609-4929-a270-684d3706028b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:22.429879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dc56018c-3609-4929-a270-684d3706028b None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:22.431468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dc56018c-3609-4929-a270-684d3706028b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:22.517913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:22.529970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1236/2455] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:22.541090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:22.542623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:22.559200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:22.611311 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0c112d-4f67-4531-9fc8-4641280481fe admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:22.615891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:22.616935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:22.617423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:22.617873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:22.635259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:22.638057 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:22.655303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1220/2456] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:46:23.023209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56018c-3609-4929-a270-684d3706028b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:23.024765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dc56018c-3609-4929-a270-684d3706028b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:23.039344 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc56018c-3609-4929-a270-684d3706028b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:23.044861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:23.045707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:23.045892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:23.046082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] PATH_INFO: `/v3/projects/6b616599cdcd4d20a9456da6296b97c0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:23.048484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b616599cdcd4d20a9456da6296b97c0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:23.052201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:23.135787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b616599cdcd4d20a9456da6296b97c0', 'request_id': 'req-afb054ae-3f52-4eb0-a102-f15942d47c46'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.147229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b616599cdcd4d20a9456da6296b97c0', 'request_id': 'req-afb054ae-3f52-4eb0-a102-f15942d47c46'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.160858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b616599cdcd4d20a9456da6296b97c0', 'request_id': 'req-afb054ae-3f52-4eb0-a102-f15942d47c46'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.178421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] The token cache is being invalidate because project 6b616599cdcd4d20a9456da6296b97c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:23.178726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.179548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.181748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1237/2457] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:23 2022] DELETE /identity/v3/projects/6b616599cdcd4d20a9456da6296b97c0 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:23.186490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:23.199343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0674f31b-8dc5-4c35-bab0-a16e833dc823 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:23.270952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:23.271593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:23.271868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:23.272138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] PATH_INFO: `/v3/users/1c15647a2b334f3bbbc10ad8cd923875` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:23.273379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c15647a2b334f3bbbc10ad8cd923875)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:23.275363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:23.335887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c15647a2b334f3bbbc10ad8cd923875'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.345350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c15647a2b334f3bbbc10ad8cd923875'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.355567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c15647a2b334f3bbbc10ad8cd923875'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.356359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c15647a2b334f3bbbc10ad8cd923875'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.357035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c15647a2b334f3bbbc10ad8cd923875'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:23.381374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1221/2458] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:23 2022] DELETE /identity/v3/users/1c15647a2b334f3bbbc10ad8cd923875 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:23.589712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:23.592218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:23.674350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb054ae-3f52-4eb0-a102-f15942d47c46 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:23.758489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:23.759127 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:23.759412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:23.759688 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:23.761712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:23.780593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:23.784053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:23.831545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:23.856981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:23.870320 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:23.871802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0b08a0-fcd6-4b05-9314-4dc4cf7e78fa service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:23.888859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1238/2459] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:23.898775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6353f482-6233-4fc1-87ce-e6b3ded619b8 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:23.909909 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:23.911575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:23.912185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:23.912843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] PATH_INFO: `/v3/projects/3efb8ddf7d0c48698b7e9f54406414ee` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:23.915250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3efb8ddf7d0c48698b7e9f54406414ee)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:23.919597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.021140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3efb8ddf7d0c48698b7e9f54406414ee', 'request_id': 'req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:24.029615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3efb8ddf7d0c48698b7e9f54406414ee', 'request_id': 'req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:24.039728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3efb8ddf7d0c48698b7e9f54406414ee', 'request_id': 'req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:24.059342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] The token cache is being invalidate because project 3efb8ddf7d0c48698b7e9f54406414ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:24.059663 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:24.060363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2c6dbe6b-7ad8-4fd2-a528-39d2e1579da9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:24.061918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1222/2460] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:23 2022] DELETE /identity/v3/projects/3efb8ddf7d0c48698b7e9f54406414ee => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:24.070741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.071625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.139065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:24.140367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:24.223411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1239/2461] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:24.231980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.232337 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.232578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.254357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:24.255627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:24.265388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1223/2462] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:24.272812 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.291668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5bb742-c142-49f0-b819-645dd3f1d4b9 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.300727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.301764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.302773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.302773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.303905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:24.304406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.306553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.326667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1240/2463] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:24.329425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.340304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093a9823-f6a5-40d3-a104-cc7b8a72635c None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.345326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.345962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.346515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.346794 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.347451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.349453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.395172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1224/2464] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/projects => generated 379 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:24.406254 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.418266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebc748f-ccc2-419a-b47d-2fc4997b3c92 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.483847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.484919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.485302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.485687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.486450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.488641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.523417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1241/2465] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/users => generated 290 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:24.528826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.548542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011d774-53f8-4be6-af62-b23092580e96 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.556416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.557430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.557926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.558420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.559618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.562767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.584302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1225/2466] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:24.588189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.600317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7341e757-f42c-4f32-9fe4-916c9e9c8b5e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.605376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b59f6d-7184-47ce-97bf-a9a241f2bd9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.606119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b59f6d-7184-47ce-97bf-a9a241f2bd9e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.606446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b59f6d-7184-47ce-97bf-a9a241f2bd9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.607107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b59f6d-7184-47ce-97bf-a9a241f2bd9e admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.608214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b59f6d-7184-47ce-97bf-a9a241f2bd9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.609290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b59f6d-7184-47ce-97bf-a9a241f2bd9e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.622140 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1242/2467] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:24.628424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.639804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1eb3c8-4aff-4b71-bc8d-4844fb566e32 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.645517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.646201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.646489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.646754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] PATH_INFO: `/v3/projects/1724d50ea3cb4db7b8b9363ad2f7198c/users/3e496ffd2de9475ca2d3a696ebe2e2d8/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.664457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1724d50ea3cb4db7b8b9363ad2f7198c, user_id=3e496ffd2de9475ca2d3a696ebe2e2d8, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.667151 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.683178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1226/2468] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:24 2022] PUT /identity/v3/projects/1724d50ea3cb4db7b8b9363ad2f7198c/users/3e496ffd2de9475ca2d3a696ebe2e2d8/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:24.701802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.702090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.702389 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.734600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] MFA Rules not processed for user `3e496ffd2de9475ca2d3a696ebe2e2d8`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:24.735829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:24.745307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:24.814518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1243/2469] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:24.819196 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.882567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:24.901874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c51669-8657-4ec2-8311-be2505b1e2b8 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:24.908352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:24.909105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:24.909491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:24.909913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:24.910657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:24.912949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:24.944430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1227/2470] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/projects => generated 381 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:24.950535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:24.963996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385aae85-bb49-4560-87e8-ae811aa38c18 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:25.039492 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:25.040135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.040706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.040870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.041495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:25.043485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:25.080356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1244/2471] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:24 2022] POST /identity/v3/users => generated 297 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:25.082836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:25.094029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e03570a-7b13-4d27-a4d0-528ed245ccfa admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:25.100092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:25.100874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.101234 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.101573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.102306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:25.104039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:25.116360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1228/2472] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:25.120865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:25.133199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9566781-37dd-407d-8251-71ae6dd77f7d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:25.137983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:25.138670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.138984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.139264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] PATH_INFO: `/v3/projects/efa1ab182faf40ccbc1da26c61353aab/users/88333a3f3c3f4585b02b6499ec574684/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.159388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efa1ab182faf40ccbc1da26c61353aab, user_id=88333a3f3c3f4585b02b6499ec574684, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:25.162227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:25.180523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1245/2473] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:25 2022] PUT /identity/v3/projects/efa1ab182faf40ccbc1da26c61353aab/users/88333a3f3c3f4585b02b6499ec574684/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:25.185409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:25.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05afbae2-ef75-41c0-af12-4a6f5d700fd1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:25.283324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dbe9d7-3f78-41f3-9913-e1bd7e3b7d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:25.284131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbe9d7-3f78-41f3-9913-e1bd7e3b7d5f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.284556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbe9d7-3f78-41f3-9913-e1bd7e3b7d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.284930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dbe9d7-3f78-41f3-9913-e1bd7e3b7d5f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.285731 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dbe9d7-3f78-41f3-9913-e1bd7e3b7d5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:25.288766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dbe9d7-3f78-41f3-9913-e1bd7e3b7d5f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:25.300289 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1229/2474] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:25 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:25.306856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:25.332965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb49ab9-ed38-44fc-9dca-b035c7d1d099 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:25.341460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:25.343262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.343738 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.344850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] PATH_INFO: `/v3/projects/efa1ab182faf40ccbc1da26c61353aab/users/88333a3f3c3f4585b02b6499ec574684/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.349372 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=efa1ab182faf40ccbc1da26c61353aab, user_id=88333a3f3c3f4585b02b6499ec574684, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:25.356061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:25.380917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1246/2475] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:25 2022] PUT /identity/v3/projects/efa1ab182faf40ccbc1da26c61353aab/users/88333a3f3c3f4585b02b6499ec574684/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:25.464631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.465086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.465539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.513675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] MFA Rules not processed for user `88333a3f3c3f4585b02b6499ec574684`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:25.515254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:25.606990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1230/2476] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:25 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:25.639030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:25.642498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:25.762245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1ec1a4-9498-4187-ba2a-74254f3ffe3f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:25.768101 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:25.768939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:25.769371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:25.769844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:25.782415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:25.784077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:25.795956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1247/2477] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:25 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:26.217121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:26.229512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d559f6f6-561f-4fc8-9389-0f5981ebfcb4 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:26.234570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '88333a3f3c3f4585b02b6499ec574684', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efa1ab182faf40ccbc1da26c61353aab', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:26.235300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:26.235588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:26.235864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:26.236579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:26.236892 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:26.238551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:26.252758 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1231/2478] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:46:26.257967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:26.264303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:26.270307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfa8d3-62a0-4925-9beb-dd00e164981e service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:26.279224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '88333a3f3c3f4585b02b6499ec574684', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efa1ab182faf40ccbc1da26c61353aab', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:26.279921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:26.280407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:26.280871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:26.282073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:26.284640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:26.326154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1248/2479] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:26 2022] POST /identity/v3/projects => generated 354 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:26.332076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:26.350215 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b98f0b-6192-4de7-99b1-337f1ea2d3b4 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:26.410699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:26.459130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '88333a3f3c3f4585b02b6499ec574684', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efa1ab182faf40ccbc1da26c61353aab', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:26.460125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:26.460605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:26.461014 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:26.462023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:26.462547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:26.466887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:26.491108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1232/2480] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:26.537010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:26.552164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f9758-e6e0-484b-9450-914ddda75637 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:26.560585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '88333a3f3c3f4585b02b6499ec574684', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efa1ab182faf40ccbc1da26c61353aab', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:26.561719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:26.562313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:26.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] PATH_INFO: `/v3/projects/c63810540f6445d4b5dd23ea532985e8` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:26.565615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorizing `identity:delete_project(project_id=c63810540f6445d4b5dd23ea532985e8)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:26.569534 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:26.648004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c63810540f6445d4b5dd23ea532985e8', 'request_id': 'req-fb93939c-93ad-4f3b-9c67-a9d466780237'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:26.655638 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c63810540f6445d4b5dd23ea532985e8', 'request_id': 'req-fb93939c-93ad-4f3b-9c67-a9d466780237'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:26.666645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c63810540f6445d4b5dd23ea532985e8', 'request_id': 'req-fb93939c-93ad-4f3b-9c67-a9d466780237'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:26.680805 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] The token cache is being invalidate because project c63810540f6445d4b5dd23ea532985e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:26.681152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:26.681875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fb93939c-93ad-4f3b-9c67-a9d466780237 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:26.683715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1249/2481] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:26 2022] DELETE /identity/v3/projects/c63810540f6445d4b5dd23ea532985e8 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:26.944082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:26.946045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:27.027918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea59897-307a-46e0-9df6-4fbb17f473d8 tempest-VolumeQuotaClassesTest-1495652529 tempest-VolumeQuotaClassesTest-1495652529-project-admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:27.095433 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:27.096092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:27.096672 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:27.096946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] PATH_INFO: `/v3/users/3e496ffd2de9475ca2d3a696ebe2e2d8` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:27.098175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e496ffd2de9475ca2d3a696ebe2e2d8)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:27.100132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:27.157768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e496ffd2de9475ca2d3a696ebe2e2d8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:27.168186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e496ffd2de9475ca2d3a696ebe2e2d8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:27.178489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e496ffd2de9475ca2d3a696ebe2e2d8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:27.179188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e496ffd2de9475ca2d3a696ebe2e2d8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:27.179987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e496ffd2de9475ca2d3a696ebe2e2d8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:27.195005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:27.205282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1233/2482] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:26 2022] DELETE /identity/v3/users/3e496ffd2de9475ca2d3a696ebe2e2d8 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:27.210802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:27.211138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:27.211476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:27.233787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:27.235370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:27.309321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1250/2483] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:27.320113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:27.321819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:27.394288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848912b7-e4fa-4c1e-a8fd-75c828420159 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:27.399633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:27.400697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:27.401180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:27.401652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:27.419041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:27.421727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:27.439219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1234/2484] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:27.974781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:27.976949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:27.997103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dea1c9-11de-46be-9df3-4c9925c2cc24 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:28.003200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:28.003881 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:28.004138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:28.004453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] PATH_INFO: `/v3/projects/1724d50ea3cb4db7b8b9363ad2f7198c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:28.005751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1724d50ea3cb4db7b8b9363ad2f7198c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:28.007720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:28.067200 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1724d50ea3cb4db7b8b9363ad2f7198c', 'request_id': 'req-d8a169b0-522e-4608-9c16-71c9ad396fd7'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.075744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1724d50ea3cb4db7b8b9363ad2f7198c', 'request_id': 'req-d8a169b0-522e-4608-9c16-71c9ad396fd7'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.081004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=84,keystone:INSERT=3,keystone:DELETE=13 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:28.091333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1724d50ea3cb4db7b8b9363ad2f7198c', 'request_id': 'req-d8a169b0-522e-4608-9c16-71c9ad396fd7'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.111445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] The token cache is being invalidate because project 1724d50ea3cb4db7b8b9363ad2f7198c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:28.112086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.113323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.115917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1251/2485] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:27 2022] DELETE /identity/v3/projects/1724d50ea3cb4db7b8b9363ad2f7198c => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:28.121551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:28.135547 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06320764-7838-4217-8a99-ac8a91d8746d service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:28.231497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:28.232159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:28.232486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:28.232833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] PATH_INFO: `/v3/users/88333a3f3c3f4585b02b6499ec574684` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:28.234029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88333a3f3c3f4585b02b6499ec574684)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:28.236132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:28.336351 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88333a3f3c3f4585b02b6499ec574684'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.355287 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88333a3f3c3f4585b02b6499ec574684'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.377961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88333a3f3c3f4585b02b6499ec574684'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.378847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88333a3f3c3f4585b02b6499ec574684'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.379613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88333a3f3c3f4585b02b6499ec574684'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:28.414051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1235/2486] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:28 2022] DELETE /identity/v3/users/88333a3f3c3f4585b02b6499ec574684 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:28.442773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:28.444273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:28.638026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a169b0-522e-4608-9c16-71c9ad396fd7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:28.823957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:28.825528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:28.827087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:28.827370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:28.833456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:28.855956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:28.857712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a23b4b-5c65-4f16-8284-3f647d178a57 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:28.870908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1252/2487] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:46:28.971565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:28.974566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:29.072819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:29.152016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bfcc06-b245-4a3a-9729-76db461d0d42 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.177104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.178549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.179523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.180315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] PATH_INFO: `/v3/projects/efa1ab182faf40ccbc1da26c61353aab` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.185640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=efa1ab182faf40ccbc1da26c61353aab)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.193750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.292958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efa1ab182faf40ccbc1da26c61353aab', 'request_id': 'req-12f19c16-6482-4371-9120-ae6d3e0a5ea9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:29.308819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efa1ab182faf40ccbc1da26c61353aab', 'request_id': 'req-12f19c16-6482-4371-9120-ae6d3e0a5ea9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:29.332712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efa1ab182faf40ccbc1da26c61353aab', 'request_id': 'req-12f19c16-6482-4371-9120-ae6d3e0a5ea9'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:29.354569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] The token cache is being invalidate because project efa1ab182faf40ccbc1da26c61353aab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:29.355527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:29.356795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-12f19c16-6482-4371-9120-ae6d3e0a5ea9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:29.359376 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1236/2488] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:28 2022] DELETE /identity/v3/projects/efa1ab182faf40ccbc1da26c61353aab => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:29.370027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.370497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.371310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.426855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:29.428449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:29.509940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1253/2489] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:29.522589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.522925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.523288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.546363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:29.548205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:29.556869 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1237/2490] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:29.563996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:29.581856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0d19d7-e7d0-4e72-b0b8-4b77666a3df0 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.588445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.589419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.589831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.590077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.590876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:29.591418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.593546 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.606245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=1,keystone:SELECT=99 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:29.613187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1254/2491] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:29.620115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:29.633506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c752c07-da54-433a-95b9-590fa5bee080 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.639118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.640063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.640537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.640956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.641836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.643979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.674494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1238/2492] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:29 2022] POST /identity/v3/projects => generated 381 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:29.679648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:29.692543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc833b7-4cac-409e-bb35-349ad7dac787 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.758540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.759165 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.759445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.759725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.760447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.762334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.787045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1255/2493] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:29 2022] POST /identity/v3/users => generated 291 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:29.791483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:29.803427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ce59cf-1988-43c3-ab3b-79aafc54259c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.808144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.808800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.809074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.809344 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.809989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.811593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.822244 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1239/2494] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:29.826486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:29.839181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562620dd-8c04-4fea-a39f-d3dfb132bb96 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.844086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347c9cb5-1c73-463b-9596-1cc0407c268b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.845300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347c9cb5-1c73-463b-9596-1cc0407c268b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.845584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347c9cb5-1c73-463b-9596-1cc0407c268b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.845860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347c9cb5-1c73-463b-9596-1cc0407c268b admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.846787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347c9cb5-1c73-463b-9596-1cc0407c268b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.848453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347c9cb5-1c73-463b-9596-1cc0407c268b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.861132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1256/2495] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:29.865861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:29.878236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e539c-5f12-4cc5-8da0-69187fc3dd1e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:29.883541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:29.884386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.884687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.884962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] PATH_INFO: `/v3/projects/3594bb56a537462d846a7540774db361/users/fb20804bacc546159fa2b66c354cf862/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.902261 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3594bb56a537462d846a7540774db361, user_id=fb20804bacc546159fa2b66c354cf862, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:29.905212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:29.922497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1240/2496] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:29 2022] PUT /identity/v3/projects/3594bb56a537462d846a7540774db361/users/fb20804bacc546159fa2b66c354cf862/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:29.935517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:29.935873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:29.936316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:29.967360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] MFA Rules not processed for user `fb20804bacc546159fa2b66c354cf862`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:29.968639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:30.040703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1257/2497] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:29 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:30.045624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.097377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:30.117069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a155d4-0f24-4fd5-9ea9-d77f9ddfecc9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.122067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.122696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.122979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.123250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.123901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.125865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.159851 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1241/2498] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:30 2022] POST /identity/v3/projects => generated 381 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:30.164512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.176487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2677ff8-b4a0-4989-b64a-7ec185190b67 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.253647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.254400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.254787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.255158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.256041 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.258952 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.287095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:30.299989 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1258/2499] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:30 2022] POST /identity/v3/users => generated 297 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:30.305132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.318203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93a83ce-a96c-4ab7-83f4-fafb3d846a45 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.323108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.323735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.324011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.324324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.325115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.326776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.341409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1242/2500] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:30.346243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.357838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4913bf86-ebfb-4f68-8181-74f73a4fea3f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.363701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.364416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.364698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.364970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] PATH_INFO: `/v3/projects/ae5db1f5ca1a4246b77dd1958eb07fa9/users/b0a484432a81421c9454bb86f7ce9131/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.382716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae5db1f5ca1a4246b77dd1958eb07fa9, user_id=b0a484432a81421c9454bb86f7ce9131, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.385526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.406446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1259/2501] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:30 2022] PUT /identity/v3/projects/ae5db1f5ca1a4246b77dd1958eb07fa9/users/b0a484432a81421c9454bb86f7ce9131/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:30.411084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.430489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8be010c-bd0b-4288-9d78-778b7fa0675c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.543540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1200c4-bfac-4078-ab84-dd4efd1cc177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.544609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1200c4-bfac-4078-ab84-dd4efd1cc177 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.545253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1200c4-bfac-4078-ab84-dd4efd1cc177 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.545542 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1200c4-bfac-4078-ab84-dd4efd1cc177 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.546705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1200c4-bfac-4078-ab84-dd4efd1cc177 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.549973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1200c4-bfac-4078-ab84-dd4efd1cc177 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.573464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1243/2502] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:30 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:30.585042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.602986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b73d137-d31d-4595-a2c0-4bb9393c3686 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.612579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.616962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.617743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.618251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] PATH_INFO: `/v3/projects/ae5db1f5ca1a4246b77dd1958eb07fa9/users/b0a484432a81421c9454bb86f7ce9131/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.623998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae5db1f5ca1a4246b77dd1958eb07fa9, user_id=b0a484432a81421c9454bb86f7ce9131, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.632396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.653773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1260/2503] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:30 2022] PUT /identity/v3/projects/ae5db1f5ca1a4246b77dd1958eb07fa9/users/b0a484432a81421c9454bb86f7ce9131/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:30.666004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.666626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.666898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.704605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] MFA Rules not processed for user `b0a484432a81421c9454bb86f7ce9131`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:30.706379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:30.797280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1244/2504] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:30 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:30.811564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:30.814238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:30.886187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f08e1b-cf2f-425f-a8b5-9e7db42e9d5c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:30.890635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:30.891386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:30.891672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:30.891946 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:30.904667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:30.906715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:30.920198 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1261/2505] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:30 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:31.061383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:31.071969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601d41dd-f6e9-4ee7-8caf-b6b862e41541 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:31.075898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e63c77-70bf-413f-9403-cf9e1171d921 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:31.076560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e63c77-70bf-413f-9403-cf9e1171d921 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:31.076834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e63c77-70bf-413f-9403-cf9e1171d921 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:31.077101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e63c77-70bf-413f-9403-cf9e1171d921 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:31.127029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:31.153339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e63c77-70bf-413f-9403-cf9e1171d921 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:31.154805 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e63c77-70bf-413f-9403-cf9e1171d921 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:31.166372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1245/2506] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:31 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:46:32.016788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:32.018420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:32.105667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e8155a-b356-4354-b292-50784364a490 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:32.113177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:32.114232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:32.114232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:32.115315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] PATH_INFO: `/v3/users/842f65f130124a01b7ff8ca3f4d5cb69` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:32.116905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=842f65f130124a01b7ff8ca3f4d5cb69)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:32.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:32.189578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '842f65f130124a01b7ff8ca3f4d5cb69'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:32.190281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '842f65f130124a01b7ff8ca3f4d5cb69'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:32.190891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '842f65f130124a01b7ff8ca3f4d5cb69'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:32.201199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '842f65f130124a01b7ff8ca3f4d5cb69'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:32.210770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '842f65f130124a01b7ff8ca3f4d5cb69'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:32.213058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:32.243257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1262/2507] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:32 2022] DELETE /identity/v3/users/842f65f130124a01b7ff8ca3f4d5cb69 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:32.248579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:32.249445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:32.249445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:32.270824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:32.272074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:32.343920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1246/2508] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:32.355510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:32.357160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:32.452876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77123f68-756c-4847-b572-131e500501c4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:32.458444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:32.459818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:32.459818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:32.459818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:32.471790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:32.473310 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:32.496951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1263/2509] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:32.868771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:32.871154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:32.893536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd59a05-8ef8-4a70-84a0-738968d9a212 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:32.903568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:32.904648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:32.905089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:32.905545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] PATH_INFO: `/v3/projects/0886880b733e4ed8841a8ce447448728` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:32.908103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0886880b733e4ed8841a8ce447448728)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:32.912099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:32.999733 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:33.023598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf91eec-752a-432b-b140-8638831a1fb0 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:33.027040 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0886880b733e4ed8841a8ce447448728', 'request_id': 'req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.040581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0886880b733e4ed8841a8ce447448728', 'request_id': 'req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.058475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0886880b733e4ed8841a8ce447448728', 'request_id': 'req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.090836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] The token cache is being invalidate because project 0886880b733e4ed8841a8ce447448728 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:33.091224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.091917 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.093786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1247/2510] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:32 2022] DELETE /identity/v3/projects/0886880b733e4ed8841a8ce447448728 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:33.098933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:33.100478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:33.129233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:33.130651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:33.131136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:33.131628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:33.136886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:33.208644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0efaa3-e4f7-49c9-ba15-e4a7041ad25d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:33.213993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:33.214716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:33.215210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:33.215534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] PATH_INFO: `/v3/users/10543d4b517b47479ac022a4f466ff10` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:33.216935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] RBAC: Authorizing `identity:delete_user(user_id=10543d4b517b47479ac022a4f466ff10)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:33.219056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:33.223447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:33.265097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:33.269153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:33.292564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1264/2511] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:46:33.299285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10543d4b517b47479ac022a4f466ff10'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.311890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10543d4b517b47479ac022a4f466ff10'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.322073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10543d4b517b47479ac022a4f466ff10'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.323244 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10543d4b517b47479ac022a4f466ff10'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.324035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d1b98287-f6ae-4074-9d7e-bea9d390349f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10543d4b517b47479ac022a4f466ff10'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:33.353564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1248/2512] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:33 2022] DELETE /identity/v3/users/10543d4b517b47479ac022a4f466ff10 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:33.808519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:33.835137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b54833-a296-4f3e-9e71-caff2945c391 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:33.911816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:33.960421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:33.961470 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:33.961942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:33.962394 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] PATH_INFO: `/v3/projects/4976d5bf279b43218ccd4cceca03b05c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:33.964465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4976d5bf279b43218ccd4cceca03b05c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:33.968043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.049153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4976d5bf279b43218ccd4cceca03b05c', 'request_id': 'req-edda1185-3575-48ff-b81b-369262343b45'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.058995 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4976d5bf279b43218ccd4cceca03b05c', 'request_id': 'req-edda1185-3575-48ff-b81b-369262343b45'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.070240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4976d5bf279b43218ccd4cceca03b05c', 'request_id': 'req-edda1185-3575-48ff-b81b-369262343b45'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.101556 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] The token cache is being invalidate because project 4976d5bf279b43218ccd4cceca03b05c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:34.102076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.103231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.106955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1265/2513] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:33 2022] DELETE /identity/v3/projects/4976d5bf279b43218ccd4cceca03b05c => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:34.113725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4b9d87-8045-4c3e-bacd-db7492e64c01 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.114174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4b9d87-8045-4c3e-bacd-db7492e64c01 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.114377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.114784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4b9d87-8045-4c3e-bacd-db7492e64c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.116304 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:34.142974 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-1d4b9d87-8045-4c3e-bacd-db7492e64c01 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:34.145150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1d4b9d87-8045-4c3e-bacd-db7492e64c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:34.241373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda1185-3575-48ff-b81b-369262343b45 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.257460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.260123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.260486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.260699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] PATH_INFO: `/v3/users/fb20804bacc546159fa2b66c354cf862` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.263802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb20804bacc546159fa2b66c354cf862)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.267420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.290644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1249/2514] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:34.305899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.306770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.306770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.336418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:34.337995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:34.349036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1250/2515] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:34.357615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.376145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae4b6ae-1b87-4292-a76b-d5cf7f33f8ef None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.386539 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.388994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.389488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.390092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.392400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:34.393170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.396384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.421808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb20804bacc546159fa2b66c354cf862'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.424759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb20804bacc546159fa2b66c354cf862'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.425883 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb20804bacc546159fa2b66c354cf862'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.432450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1251/2516] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:34.436863 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.439259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:34.448787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb20804bacc546159fa2b66c354cf862'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.456552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56b0067-0c03-4c98-896b-afd26094429d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.464160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.464987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.465278 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.465742 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.466629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.468811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.476109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-bb16b149-3186-48be-82a8-445108991148 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb20804bacc546159fa2b66c354cf862'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:34.490876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:34.508303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1252/2517] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:34 2022] POST /identity/v3/projects => generated 379 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:34.514508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.529872 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1266/2518] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:34 2022] DELETE /identity/v3/users/fb20804bacc546159fa2b66c354cf862 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:34.539440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.540161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.540680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.581567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:34.583922 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:34.601812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:34.649355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e84398c-9810-4a6c-b962-c6f7b7cf58f4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.709465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1267/2519] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:34.723968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.726582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:34.791122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.792176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.792902 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.793412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.794815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.799138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.856883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1253/2520] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:34 2022] POST /identity/v3/users => generated 290 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:34.859341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.890029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2203d9-75a3-4675-8cf9-e866f56f6aac admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.894934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa440d7-799a-428a-8abf-febecb656ebc None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.899624 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.900373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.900727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.901073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.901829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.903815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.916704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.917249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1254/2521] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:34 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:34.924274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:34.924728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.925256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.925715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.950021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ee3e73-dcc6-4705-954c-fad45c93e7e5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:34.956317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.958484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.963364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c1db0d-6ef6-44d7-bf58-d2891b1fd315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:34.964606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c1db0d-6ef6-44d7-bf58-d2891b1fd315 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:34.965213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c1db0d-6ef6-44d7-bf58-d2891b1fd315 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:34.965808 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c1db0d-6ef6-44d7-bf58-d2891b1fd315 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:34.967152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c1db0d-6ef6-44d7-bf58-d2891b1fd315 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:34.971316 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c1db0d-6ef6-44d7-bf58-d2891b1fd315 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:34.993682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1268/2522] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:34.996685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1255/2523] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:34 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:35.003908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.027936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05baee8-3e5f-4f80-a62c-61c84264a677 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:35.054364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:35.055611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.056119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.056677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] PATH_INFO: `/v3/projects/af7feeb1d2094f37b8bc81d00f5b5724/users/1e457a65bcd44da5b1fe045f4da7bf21/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.082569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af7feeb1d2094f37b8bc81d00f5b5724, user_id=1e457a65bcd44da5b1fe045f4da7bf21, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:35.086411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:35.110183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1269/2524] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:35 2022] PUT /identity/v3/projects/af7feeb1d2094f37b8bc81d00f5b5724/users/1e457a65bcd44da5b1fe045f4da7bf21/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:35.120792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.121080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.121357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.160022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] MFA Rules not processed for user `1e457a65bcd44da5b1fe045f4da7bf21`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:35.161841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:35.248299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1256/2525] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:35 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:35.254208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.370117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9a92f3-5edb-42e1-930b-ef9103b47668 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:35.377882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:35.378836 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.379238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.379623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.380583 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:35.383536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:35.398709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:35.450998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1270/2526] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:35 2022] POST /identity/v3/projects => generated 381 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:35.455201 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.473839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebce44d9-6705-4fc3-bda3-8465328b1dd0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:35.531248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.535216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:35.575206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:35.605668 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:35.607763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.608510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.608850 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.609632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:35.615024 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:35.678331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1257/2527] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:35 2022] POST /identity/v3/users => generated 297 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:35.682361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.715749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6dfbe5-dc31-4744-8a4f-223bf6e11899 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:35.730855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:35.731889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.733507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.734373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.736407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:35.740748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:35.746247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd845f-d830-4f28-8592-61bc424e7d5c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:35.758342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:35.760269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.761199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.761940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] PATH_INFO: `/v3/projects/3594bb56a537462d846a7540774db361` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.769951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3594bb56a537462d846a7540774db361)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:35.778179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:35.778474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1258/2528] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:35 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:35.785111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.816580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ae9b9-b1ac-40bb-a81a-8e3da3ca37be admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:35.829010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:35.830717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:35.831363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:35.831936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] PATH_INFO: `/v3/projects/96c014db070846f6868098d006c34ffd/users/c5cbdb7b27d84c7d831602774b1118db/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:35.870687 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96c014db070846f6868098d006c34ffd, user_id=c5cbdb7b27d84c7d831602774b1118db, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:35.877349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:35.919785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1259/2529] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:35 2022] PUT /identity/v3/projects/96c014db070846f6868098d006c34ffd/users/c5cbdb7b27d84c7d831602774b1118db/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:35.924169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3594bb56a537462d846a7540774db361', 'request_id': 'req-3d472903-6e94-4917-b7f3-735a8cf3851c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:35.926123 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:35.939072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3594bb56a537462d846a7540774db361', 'request_id': 'req-3d472903-6e94-4917-b7f3-735a8cf3851c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:35.955876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3594bb56a537462d846a7540774db361', 'request_id': 'req-3d472903-6e94-4917-b7f3-735a8cf3851c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.016548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] The token cache is being invalidate because project 3594bb56a537462d846a7540774db361 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:36.017069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.022431 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.027911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1271/2530] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:35 2022] DELETE /identity/v3/projects/3594bb56a537462d846a7540774db361 => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:36.034708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:36.046387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:36.121372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898770c2-3980-405b-bb76-daa9aea56bd2 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:36.130874 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:36.132464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:36.133249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:36.133963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:36.135530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:36.139804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:36.176612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1260/2531] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:35 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:36.184814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:36.192524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:36.234161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fa816c-c809-4ce8-904f-e369b42631b6 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:36.236479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d472903-6e94-4917-b7f3-735a8cf3851c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:36.245259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e93a1-375e-4bd8-8fce-dd277d0ef947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:36.247114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e93a1-375e-4bd8-8fce-dd277d0ef947 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:36.247880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e93a1-375e-4bd8-8fce-dd277d0ef947 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:36.248425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e93a1-375e-4bd8-8fce-dd277d0ef947 admin admin] PATH_INFO: `/v3/projects/96c014db070846f6868098d006c34ffd/users/c5cbdb7b27d84c7d831602774b1118db/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:36.254250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8e93a1-375e-4bd8-8fce-dd277d0ef947 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96c014db070846f6868098d006c34ffd, user_id=c5cbdb7b27d84c7d831602774b1118db, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:36.255447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:36.256512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:36.257029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:36.257328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8e93a1-375e-4bd8-8fce-dd277d0ef947 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:36.257693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] PATH_INFO: `/v3/users/b0a484432a81421c9454bb86f7ce9131` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:36.260204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0a484432a81421c9454bb86f7ce9131)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:36.263668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:36.271244 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1261/2532] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:36 2022] PUT /identity/v3/projects/96c014db070846f6868098d006c34ffd/users/c5cbdb7b27d84c7d831602774b1118db/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:36.285348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:36.285348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:36.285348 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:36.331814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:36.349434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] MFA Rules not processed for user `c5cbdb7b27d84c7d831602774b1118db`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:36.353003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:36.394683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0a484432a81421c9454bb86f7ce9131'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.396596 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0a484432a81421c9454bb86f7ce9131'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.397953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0a484432a81421c9454bb86f7ce9131'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.415224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0a484432a81421c9454bb86f7ce9131'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.435828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0a484432a81421c9454bb86f7ce9131'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:36.485965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1272/2533] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:36 2022] DELETE /identity/v3/users/b0a484432a81421c9454bb86f7ce9131 => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:36.510375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1262/2534] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:36 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:36.531244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:36.533597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:36.625161 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7651dce1-61d0-482d-8450-8478c680b026 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:36.633575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:36.635366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:36.635857 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:36.636526 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:36.771965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:36.775243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:36.796644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1273/2535] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:36 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:37.460875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:37.464821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:37.479037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:37.592540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:37.594877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:37.618894 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab39aab-f5a8-494c-b935-77aaa19a8cdd None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:37.743511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:37.750103 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:37.751620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:37.752424 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:37.752898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] PATH_INFO: `/v3/users/1e457a65bcd44da5b1fe045f4da7bf21` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:37.755576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e457a65bcd44da5b1fe045f4da7bf21)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:37.758860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:37.793554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b094346c-c27a-4bb6-901a-75d49ba1dc5f service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:37.811668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:37.813502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:37.814272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:37.814905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] PATH_INFO: `/v3/projects/ae5db1f5ca1a4246b77dd1958eb07fa9` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:37.817998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae5db1f5ca1a4246b77dd1958eb07fa9)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:37.822138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:37.896390 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e457a65bcd44da5b1fe045f4da7bf21'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.916412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e457a65bcd44da5b1fe045f4da7bf21'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.944256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae5db1f5ca1a4246b77dd1958eb07fa9', 'request_id': 'req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.944505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e457a65bcd44da5b1fe045f4da7bf21'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.945990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e457a65bcd44da5b1fe045f4da7bf21'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.947507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-3c1f1836-e2ff-4990-b934-babe6413e516 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e457a65bcd44da5b1fe045f4da7bf21'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.961752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae5db1f5ca1a4246b77dd1958eb07fa9', 'request_id': 'req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:37.983182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae5db1f5ca1a4246b77dd1958eb07fa9', 'request_id': 'req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:38.004304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1263/2536] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:37 2022] DELETE /identity/v3/users/1e457a65bcd44da5b1fe045f4da7bf21 => generated 0 bytes in 543 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:38.011032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.011639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.012159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.024772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] The token cache is being invalidate because project ae5db1f5ca1a4246b77dd1958eb07fa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:38.025199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:38.026318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-65ab9a5b-b771-4920-8dad-7ad0fec6e9d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:38.029239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1274/2537] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:37 2022] DELETE /identity/v3/projects/ae5db1f5ca1a4246b77dd1958eb07fa9 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:38.037915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10bfba1-7933-4272-97ba-3529f1ee6e5d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.038359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10bfba1-7933-4272-97ba-3529f1ee6e5d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.038858 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10bfba1-7933-4272-97ba-3529f1ee6e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.050034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:38.052061 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:38.065201 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d10bfba1-7933-4272-97ba-3529f1ee6e5d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:38.067477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d10bfba1-7933-4272-97ba-3529f1ee6e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:38.125005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:38.201023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1275/2538] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:38.223517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.225118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.225589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.234599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1264/2539] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:38.250632 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:38.256080 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:38.291212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:38.295747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:38.328329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1276/2540] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:38.334874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:38.363656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d994f9-3c7f-4d5c-b5a6-f19958781819 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:38.376996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:38.377839 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.378193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.378458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.379048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6e01a5-8159-4c7f-93e0-5c22e9158107 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:38.379264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:38.379454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:38.381540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:38.388652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:38.389565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.390034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.390409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.404984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1277/2541] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:38.405322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:38.406082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:38.407459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:38.423409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1265/2542] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:38.424729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f77a2cc-2c6a-4ee2-af1d-1bf414058f7d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:38.430862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:38.431798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.432169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.432654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.433370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:38.435701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:38.513315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1278/2543] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:38 2022] POST /identity/v3/projects => generated 385 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:38.518393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:38.540689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cb4444-16ca-426b-8456-556611209f18 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:38.705402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:38.708639 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.709427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.710065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:38.730306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:38.834651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1266/2544] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:38 2022] POST /identity/v3/users => generated 293 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:38.840214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:38.864226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd821a09-2171-4985-aaba-b448dff77aed admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:38.875105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:38.876731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.877265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.877708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.879156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:38.883246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:38.911187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1279/2545] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:38 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:38.917463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:38.942958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08695359-3cf6-4c4a-8f0e-daa39ace195a admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:38.952154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:38.953384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:38.955733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:38.956730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:38.958780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:38.962483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:38.996314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1267/2546] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:38 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:39.000951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.028543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e684e7-488b-4d7d-9fc5-23165884187b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.048335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2238d28-4147-43e0-b692-ccaa4f6386e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:39.049257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2238d28-4147-43e0-b692-ccaa4f6386e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.049696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2238d28-4147-43e0-b692-ccaa4f6386e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.050197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2238d28-4147-43e0-b692-ccaa4f6386e8 admin admin] PATH_INFO: `/v3/projects/fb68c2d1eb3c437798cbb2828591cf02/users/ba24df9dc4c94ccd96d0d977a3aa529f/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.111364 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.114477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.119622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2238d28-4147-43e0-b692-ccaa4f6386e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb68c2d1eb3c437798cbb2828591cf02, user_id=ba24df9dc4c94ccd96d0d977a3aa529f, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:39.133935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2238d28-4147-43e0-b692-ccaa4f6386e8 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:39.175629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1280/2547] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:38 2022] PUT /identity/v3/projects/fb68c2d1eb3c437798cbb2828591cf02/users/ba24df9dc4c94ccd96d0d977a3aa529f/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:39.181955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff34f89-442e-4ba6-9990-04d2c95938b7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.194923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.195300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.195774 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.224095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:39.298155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] MFA Rules not processed for user `ba24df9dc4c94ccd96d0d977a3aa529f`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:39.300533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.356377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:39.368049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:39.369116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.369582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.370037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] PATH_INFO: `/v3/projects/af7feeb1d2094f37b8bc81d00f5b5724` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.372004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af7feeb1d2094f37b8bc81d00f5b5724)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:39.376517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:39.491899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af7feeb1d2094f37b8bc81d00f5b5724', 'request_id': 'req-edb1c41e-3069-42f1-bd32-10656e930812'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.501416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1281/2548] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:39.503006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.503933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af7feeb1d2094f37b8bc81d00f5b5724', 'request_id': 'req-edb1c41e-3069-42f1-bd32-10656e930812'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.504510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.520684 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af7feeb1d2094f37b8bc81d00f5b5724', 'request_id': 'req-edb1c41e-3069-42f1-bd32-10656e930812'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.544419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] The token cache is being invalidate because project af7feeb1d2094f37b8bc81d00f5b5724 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:39.545082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.546225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.549137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1268/2549] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:39 2022] DELETE /identity/v3/projects/af7feeb1d2094f37b8bc81d00f5b5724 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:39.550325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.553747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.604523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee6f33c-0c96-4697-87a0-b749bd719479 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.656971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb1c41e-3069-42f1-bd32-10656e930812 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.663521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:39.664462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.665541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.666481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] PATH_INFO: `/v3/users/ec931c9cc56a462991413926d34dfa46` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.668992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec931c9cc56a462991413926d34dfa46)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:39.672431 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:39.696044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:39.696730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.697010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.697285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.699253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.715046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:39.716903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:39.733664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1282/2550] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:39.734569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.736798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.750791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec931c9cc56a462991413926d34dfa46'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.760523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a5a323-30bf-4b5a-be51-0c96dce6551e service placement] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.764516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec931c9cc56a462991413926d34dfa46'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.770393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:39.771474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.772108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.772715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.774427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:39.778591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:39.779873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec931c9cc56a462991413926d34dfa46'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.781180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec931c9cc56a462991413926d34dfa46'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.782225 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec931c9cc56a462991413926d34dfa46'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:39.808000 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1269/2551] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:39 2022] DELETE /identity/v3/users/ec931c9cc56a462991413926d34dfa46 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:39.808831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.810393 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.820604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:39.832002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1283/2552] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:39 2022] POST /identity/v3/projects => generated 385 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:39.837021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:39.840014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:39.849762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:39.922946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349f16ce-cc36-49f9-9305-5694859c93ac admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.973052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240700de-69bb-416d-abb1-4ed34b4a2d80 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:39.982644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:39.983620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:39.984086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:39.984551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:39.986384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:39.991453 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.023983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.025583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.026365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.026916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] PATH_INFO: `/v3/users/c5cbdb7b27d84c7d831602774b1118db` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.029391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5cbdb7b27d84c7d831602774b1118db)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.030877 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1284/2553] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:39 2022] POST /identity/v3/users => generated 299 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:40.035901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.037203 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.053374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56497d0-6988-46c5-af2d-e3a9867f5d63 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.061440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.062668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.063357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.063936 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.065440 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.067982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.088552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1285/2554] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:40 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:40.093491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.109604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac10ed5-9d9c-4c23-8f09-42bfc6e78a57 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.118095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.120692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.121970 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.124407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] PATH_INFO: `/v3/projects/142e17cb5a164d5cb7423b813d49b30c/users/211a8dc8ada44fbe9a3155ba651c7008/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.145218 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5cbdb7b27d84c7d831602774b1118db'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.157545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=142e17cb5a164d5cb7423b813d49b30c, user_id=211a8dc8ada44fbe9a3155ba651c7008, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.160444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5cbdb7b27d84c7d831602774b1118db'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.165273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.176932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5cbdb7b27d84c7d831602774b1118db'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.177737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5cbdb7b27d84c7d831602774b1118db'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.178370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5cbdb7b27d84c7d831602774b1118db'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.185859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1286/2555] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:40 2022] PUT /identity/v3/projects/142e17cb5a164d5cb7423b813d49b30c/users/211a8dc8ada44fbe9a3155ba651c7008/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:40.187096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.191524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.210069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1270/2556] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:39 2022] DELETE /identity/v3/users/c5cbdb7b27d84c7d831602774b1118db => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:40.211682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.214008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.294669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:40.320924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf3bb92-0502-4b85-a81e-a54959383ac6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.368010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97deabba-b359-49bf-b70b-9f0a33f50ac1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.375940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.377298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.377962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.378415 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.379601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.382049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.398494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1271/2557] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:40 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:40.402922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.420605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6934a42e-2eee-4bee-9734-cf1002165c6f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.431497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2515be44-88bd-456e-86fa-52f4b10c6449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.433077 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515be44-88bd-456e-86fa-52f4b10c6449 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.433345 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515be44-88bd-456e-86fa-52f4b10c6449 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.433657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515be44-88bd-456e-86fa-52f4b10c6449 admin admin] PATH_INFO: `/v3/projects/142e17cb5a164d5cb7423b813d49b30c/users/211a8dc8ada44fbe9a3155ba651c7008/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.437418 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2515be44-88bd-456e-86fa-52f4b10c6449 admin admin] RBAC: Authorizing `identity:create_grant(project_id=142e17cb5a164d5cb7423b813d49b30c, user_id=211a8dc8ada44fbe9a3155ba651c7008, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.438283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.439047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.439433 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.439969 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] PATH_INFO: `/v3/projects/9badaa18dcf64810b10dd7bf03d8e40d` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.441519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=9badaa18dcf64810b10dd7bf03d8e40d)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.442960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2515be44-88bd-456e-86fa-52f4b10c6449 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.444340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.455420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:40.468329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1272/2558] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:40 2022] PUT /identity/v3/projects/142e17cb5a164d5cb7423b813d49b30c/users/211a8dc8ada44fbe9a3155ba651c7008/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:40.483739 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878df380-3385-432d-a702-162a874b838e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.484227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878df380-3385-432d-a702-162a874b838e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.488451 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878df380-3385-432d-a702-162a874b838e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.561580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-878df380-3385-432d-a702-162a874b838e None None] MFA Rules not processed for user `211a8dc8ada44fbe9a3155ba651c7008`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:40.566153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-878df380-3385-432d-a702-162a874b838e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.573577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9badaa18dcf64810b10dd7bf03d8e40d', 'request_id': 'req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.583792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9badaa18dcf64810b10dd7bf03d8e40d', 'request_id': 'req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.596170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9badaa18dcf64810b10dd7bf03d8e40d', 'request_id': 'req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.610945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] The token cache is being invalidate because project 9badaa18dcf64810b10dd7bf03d8e40d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:40.611347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.612152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:40.625332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1287/2559] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:40 2022] DELETE /identity/v3/projects/9badaa18dcf64810b10dd7bf03d8e40d => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:40.626610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.628274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.692697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1273/2560] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:40 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:40.695328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.698062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.698999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.744358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:40.746716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.771859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2602e5-f764-47fc-b41f-34dc23f9cb6e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.777717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.778566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.778895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.779270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.782309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.863437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1274/2561] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:40.865700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.867845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.870850 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.872939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.886359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1288/2562] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:40.887446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:40.889179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.906426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0001728-6428-4f55-bd29-148732914dea service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.916404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.917521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.917996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.918711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] PATH_INFO: `/v3/projects/96c014db070846f6868098d006c34ffd` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.920947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] RBAC: Authorizing `identity:delete_project(project_id=96c014db070846f6868098d006c34ffd)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:40.924036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:40.934271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:40.967319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5c54a2-7ccc-47c7-a7cc-1d6736db74c0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:40.975576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:40.976781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:40.977317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:40.978013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:40.981249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:40.993633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:41.007234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96c014db070846f6868098d006c34ffd', 'request_id': 'req-a3971062-10a2-4e2b-ae02-c8d4b887d144'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:41.021627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96c014db070846f6868098d006c34ffd', 'request_id': 'req-a3971062-10a2-4e2b-ae02-c8d4b887d144'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:41.034727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96c014db070846f6868098d006c34ffd', 'request_id': 'req-a3971062-10a2-4e2b-ae02-c8d4b887d144'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:41.061150 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] The token cache is being invalidate because project 96c014db070846f6868098d006c34ffd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:41.061635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:41.062742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-a3971062-10a2-4e2b-ae02-c8d4b887d144 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:41.070900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1289/2563] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:40 2022] DELETE /identity/v3/projects/96c014db070846f6868098d006c34ffd => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:41.070900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.070900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.070900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.103115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:41.105266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:41.110697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.113239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca68518c-59d3-4dd3-ac71-6075a5334a15 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.130072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1275/2564] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:40 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:41.132446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c43fba-6ec3-42cd-b137-ebd1ec130dba None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.132944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c43fba-6ec3-42cd-b137-ebd1ec130dba None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.133373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c43fba-6ec3-42cd-b137-ebd1ec130dba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.173773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-92c43fba-6ec3-42cd-b137-ebd1ec130dba None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:41.176625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-92c43fba-6ec3-42cd-b137-ebd1ec130dba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:41.248549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1290/2565] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:41.253162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.286069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6557f08-b071-417a-9272-7976b38bc3d3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.292616 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.293313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.293598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.293874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.294771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:41.301882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.301882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.337497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1276/2566] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:41.340808 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1291/2567] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:41.350879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.352374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.352925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.354048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.377271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a326c41-1254-4fe3-a018-87357a5b42da admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.380419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:41.381771 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:41.389800 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.391923 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.393361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1277/2568] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:41.394554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.396324 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.397299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.399275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.403605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.409747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57f1808-5c63-43a4-a2b6-f7006b5b10de None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.416194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.417052 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.417435 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.417833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.418770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:41.419150 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.421428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.440109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1278/2569] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:41.441735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.460370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4879c225-ff3c-4d1f-a7c5-5031abc94319 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.461224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1292/2570] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:41.462700 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.545695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.546887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.547533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.548121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.549406 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.553996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.568322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d280aac-e0da-4036-ab55-9b7c5f38b175 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.577956 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.579730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.580427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.581026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.582714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.588039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.614349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:41.632108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1293/2571] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/users => generated 290 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:41.637368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.642986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1279/2572] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/projects => generated 375 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:41.655048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.670605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8652f4de-008e-47c8-a044-5d867a3fa308 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.724347 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:41.811860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd71a9-37b1-4963-bb9c-ee0e6a1e8bf5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.822810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.824123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.824695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.826444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.829659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.835326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.836471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.839423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.840721 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.841324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.842893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.853190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.875879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1294/2573] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:41 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:41.881957 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.886284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:41.909772 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1280/2574] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:41 2022] POST /identity/v3/users => generated 288 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:41.911741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.928227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc63cce-1926-4505-a522-e94f2a904e60 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.934402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.935076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.935581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.935871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.936628 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.938822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:41.952313 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1281/2575] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:41 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:41.956548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:41.956548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:41.970823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91545a42-37a9-43d3-8948-70ef72851080 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:41.976398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:41.977071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:41.977357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:41.977638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] PATH_INFO: `/v3/users/6dd0ed2144164190b380ed58b3dd7cd4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:41.978970 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6dd0ed2144164190b380ed58b3dd7cd4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:41.981088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.041471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a443745-9be3-4de2-a509-6f2d64eacfb0 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.048958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.049598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.049896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.050139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.052019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.073622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6dd0ed2144164190b380ed58b3dd7cd4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.086282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6dd0ed2144164190b380ed58b3dd7cd4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.099911 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6dd0ed2144164190b380ed58b3dd7cd4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.100494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6dd0ed2144164190b380ed58b3dd7cd4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.101525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6dd0ed2144164190b380ed58b3dd7cd4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.115075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.122486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.126866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1282/2576] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:41 2022] DELETE /identity/v3/users/6dd0ed2144164190b380ed58b3dd7cd4 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:42.127792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.129363 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.149554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1295/2577] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:42.162568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.190519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4b0a3d-21bd-4d4d-bd57-3831b4b53593 service placement] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.255034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:42.295582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e687832-5999-424a-a8cd-7a99cfe51ad3 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.309484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.310179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.310463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.310852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.311478 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.313948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.326886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1283/2578] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:42 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:42.329215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.330287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.330668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.331186 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] PATH_INFO: `/v3/projects/06ffab0f61eb42f1b6f2cefe3d126a03/users/64d5f1c8de2e44c19962957dcfd9aebf/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.333442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.351857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93549af-eea9-4a39-b4ca-b8d126215495 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.354592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06ffab0f61eb42f1b6f2cefe3d126a03, user_id=64d5f1c8de2e44c19962957dcfd9aebf, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.358489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.360936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d05d2cb-5a34-41d7-8f5d-e2765a1a9029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.362090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d05d2cb-5a34-41d7-8f5d-e2765a1a9029 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.362652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d05d2cb-5a34-41d7-8f5d-e2765a1a9029 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.363232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d05d2cb-5a34-41d7-8f5d-e2765a1a9029 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.364518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d05d2cb-5a34-41d7-8f5d-e2765a1a9029 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.367833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d05d2cb-5a34-41d7-8f5d-e2765a1a9029 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.396387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1296/2579] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:42 2022] PUT /identity/v3/projects/06ffab0f61eb42f1b6f2cefe3d126a03/users/64d5f1c8de2e44c19962957dcfd9aebf/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:42.396603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1284/2580] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:42 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:42.396734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.400669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067d45a-b037-44f4-a820-9c310bbf8ca9 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.401449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067d45a-b037-44f4-a820-9c310bbf8ca9 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.401916 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067d45a-b037-44f4-a820-9c310bbf8ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.444926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:42.461839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a067d45a-b037-44f4-a820-9c310bbf8ca9 None None] MFA Rules not processed for user `64d5f1c8de2e44c19962957dcfd9aebf`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:42.463849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a067d45a-b037-44f4-a820-9c310bbf8ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.498164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4774f8bd-81d0-4bfc-b6d0-899a70a63436 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.504841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.505621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.505902 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.506712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] PATH_INFO: `/v3/projects/b3a88556db5848d287847eed9bf8b0e1/users/a4046b5c5e08451e9f26e882b6c69893/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.527648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3a88556db5848d287847eed9bf8b0e1, user_id=a4046b5c5e08451e9f26e882b6c69893, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.531055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.551524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1297/2581] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:42 2022] PUT /identity/v3/projects/b3a88556db5848d287847eed9bf8b0e1/users/a4046b5c5e08451e9f26e882b6c69893/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:42.552468 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.561914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.576079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1285/2582] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:42.580483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.580483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.580483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.608785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] MFA Rules not processed for user `a4046b5c5e08451e9f26e882b6c69893`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:42.610579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.693660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1286/2583] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:42 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:42.694425 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.695903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.811765 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:42.820924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa41f35-4f4c-4ac3-9d6a-f5cf67eb7290 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.827238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.827963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.828548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.829430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.830322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.832580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.849318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9e3a1-0e1d-48df-8ebf-b57195d4f5cb admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:42.857170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:42.858154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:42.858597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:42.859053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] PATH_INFO: `/v3/projects/f7b08cc6d4a34961bdf464b58946a02e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:42.879382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7b08cc6d4a34961bdf464b58946a02e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:42.880538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1287/2584] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:42 2022] POST /identity/v3/projects => generated 375 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:42.881692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.882623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:42.886748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:42.951678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7b08cc6d4a34961bdf464b58946a02e', 'request_id': 'req-dce9e74b-f417-4e1b-aa5d-4570e469b35d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.957741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7b08cc6d4a34961bdf464b58946a02e', 'request_id': 'req-dce9e74b-f417-4e1b-aa5d-4570e469b35d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.968299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7b08cc6d4a34961bdf464b58946a02e', 'request_id': 'req-dce9e74b-f417-4e1b-aa5d-4570e469b35d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.987512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] The token cache is being invalidate because project f7b08cc6d4a34961bdf464b58946a02e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:42.987990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.989058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:42.991632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1298/2585] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:42 2022] DELETE /identity/v3/projects/f7b08cc6d4a34961bdf464b58946a02e => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:42.993070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:42.995060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:43.003068 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe571eb6-c8b8-4581-b882-ef6de546ca20 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.052709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:43.088136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.089164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.089553 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.089924 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.092395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:43.095029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce9e74b-f417-4e1b-aa5d-4570e469b35d admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.102665 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c47a60f-9059-40a6-a5fe-91c7115ba9ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.103612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c47a60f-9059-40a6-a5fe-91c7115ba9ab admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.103981 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c47a60f-9059-40a6-a5fe-91c7115ba9ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.104476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c47a60f-9059-40a6-a5fe-91c7115ba9ab admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.105520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c47a60f-9059-40a6-a5fe-91c7115ba9ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.108978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c47a60f-9059-40a6-a5fe-91c7115ba9ab admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.145997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1299/2586] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:42 2022] POST /identity/v3/users => generated 288 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:43.148478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.148963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.178714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:43.180095 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:43.193541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.193856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1300/2587] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:43.195016 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3bfdc1-76a0-4485-8421-6865604f34ca service glance] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.195272 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.205955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1288/2588] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:43.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.207422 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.207698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.213886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe463be9-d733-4c7c-83c9-11e6b49ac8d3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.239373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.239373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.239373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.239373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.240114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:43.240332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.240507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:43.242496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.248557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1289/2589] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:43.253891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.264738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac3c58-bd3b-497e-88b4-e8877cdaf2df None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.271599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.272383 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.272666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.272942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.273749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:43.274056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.275653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1301/2590] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:43 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:43.276066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.278498 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.296483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1290/2591] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:43.298007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.307485 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e7bdd-0d88-40e1-a4b4-a0af8ca2d93c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.320797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.322192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.322931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.323824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.325582 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3fd72c-59b6-4629-936e-72c11dc2c694 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.325864 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.330739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.342816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.343944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.344552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.345477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.352690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.354810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.367781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1302/2592] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:43 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:43.368122 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:43.372855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.386870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ad7748-d585-499e-a2a3-3c5bef01e19a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.395867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471b973-fcb7-4bcd-9f8b-33a59c790c8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.398213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471b973-fcb7-4bcd-9f8b-33a59c790c8c admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.398213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471b973-fcb7-4bcd-9f8b-33a59c790c8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.398213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471b973-fcb7-4bcd-9f8b-33a59c790c8c admin admin] PATH_INFO: `/v3/projects/1c8aad54ff534f8a86d76c5300eb6bbc/users/e5f53ef1c4f943b99e12dbf1a104e7db/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.420030 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471b973-fcb7-4bcd-9f8b-33a59c790c8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c8aad54ff534f8a86d76c5300eb6bbc, user_id=e5f53ef1c4f943b99e12dbf1a104e7db, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.423192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1291/2593] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/projects => generated 389 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:43.426047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471b973-fcb7-4bcd-9f8b-33a59c790c8c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.427285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.473482 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1303/2594] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:43 2022] PUT /identity/v3/projects/1c8aad54ff534f8a86d76c5300eb6bbc/users/e5f53ef1c4f943b99e12dbf1a104e7db/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:43.492702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.493136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.493580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.532613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fbf9ee-c4fb-4548-8f2b-29a841f93800 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.552732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] MFA Rules not processed for user `e5f53ef1c4f943b99e12dbf1a104e7db`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:43.555234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:43.650713 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.651977 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.652596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.653127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.654251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.657419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.679910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1304/2595] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:43.686960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.708566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1292/2596] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/users => generated 295 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:43.709868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.717462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97448f7c-37b9-4be4-9b0c-d9cf6b584006 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.727829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47a268e-3bc8-4d49-8cbf-4996fdafd27b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.728387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.729306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.729763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.730219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.731377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.733729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.734355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.735174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.735418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.735592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.736062 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.737737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.762886 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1293/2597] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:43 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:43.768268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.794497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1305/2598] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/projects => generated 379 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:43.795606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eb96f4-abc4-4d9b-8197-1fc25d1ba49e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.802224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:43.916082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:43.959590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.960968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.961604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.962186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.963156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561fa24c-ecca-4b37-814f-f5865f556da5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:43.964392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:43.970895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:43.993288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:43.994549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:43.995600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:43.996475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:43.997587 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1294/2599] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:43 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:43.998189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.002790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.003260 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.028614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72aa9e6-416e-40e2-a573-d28277b72169 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.032290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f4541-985d-4ae5-97f4-de8cd9b65043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.040042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f4541-985d-4ae5-97f4-de8cd9b65043 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.040042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f4541-985d-4ae5-97f4-de8cd9b65043 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.040042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f4541-985d-4ae5-97f4-de8cd9b65043 admin admin] PATH_INFO: `/v3/projects/fb107a3b551d4161a6eef6640ec23606/users/ef0027d9b42b495f8a0e4c546c809b3d/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.046359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:44.065950 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f4541-985d-4ae5-97f4-de8cd9b65043 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb107a3b551d4161a6eef6640ec23606, user_id=ef0027d9b42b495f8a0e4c546c809b3d, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.068037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1306/2600] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:43 2022] POST /identity/v3/users => generated 296 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:44.076652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f4541-985d-4ae5-97f4-de8cd9b65043 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.088396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.106903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1295/2601] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:44 2022] PUT /identity/v3/projects/fb107a3b551d4161a6eef6640ec23606/users/ef0027d9b42b495f8a0e4c546c809b3d/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:44.116647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72e29cb-d503-4b36-9f95-2cd5a747d6ef admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.120333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.121904 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.122696 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.181100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] MFA Rules not processed for user `ef0027d9b42b495f8a0e4c546c809b3d`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:44.182442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:44.263100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.264218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.264633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.265046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.266346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.269033 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.288687 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1307/2602] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:44 2022] GET /identity/v3/roles => generated 1420 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:44.293517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685e72c-836d-4739-bd2a-fd05d580cd3a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.322088 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.323282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.323827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.324318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] PATH_INFO: `/v3/projects/9e054f77448b4211a88e1827774ec000/users/d9fc2512816d445594144a5bf55784ac/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.344900 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1296/2603] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:44 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:44.351704 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e054f77448b4211a88e1827774ec000, user_id=d9fc2512816d445594144a5bf55784ac, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.354695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.362750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.364987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:44.376668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1308/2604] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:44 2022] PUT /identity/v3/projects/9e054f77448b4211a88e1827774ec000/users/d9fc2512816d445594144a5bf55784ac/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:44.383742 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.456065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4364d848-7d77-4034-9457-5dbfdf30424e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.485913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494b3e7-1ec8-4d1c-81b4-8c42cab9be75 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.495407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.496620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.497408 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.497838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.499645 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.503692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.522654 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1309/2605] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:44 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:44.528082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.541037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf93558-82dd-4a2d-982e-cad67994de1c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.546925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.547882 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef10bf0-fcc7-4d81-bec2-3e81db51425d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.548413 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef10bf0-fcc7-4d81-bec2-3e81db51425d admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.548724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef10bf0-fcc7-4d81-bec2-3e81db51425d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.548997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef10bf0-fcc7-4d81-bec2-3e81db51425d admin admin] PATH_INFO: `/v3/projects/9e054f77448b4211a88e1827774ec000/users/d9fc2512816d445594144a5bf55784ac/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.549846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.550172 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.550565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.551561 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef10bf0-fcc7-4d81-bec2-3e81db51425d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e054f77448b4211a88e1827774ec000, user_id=d9fc2512816d445594144a5bf55784ac, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.556748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef10bf0-fcc7-4d81-bec2-3e81db51425d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.576842 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1310/2606] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:44 2022] PUT /identity/v3/projects/9e054f77448b4211a88e1827774ec000/users/d9fc2512816d445594144a5bf55784ac/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:44.589601 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.589887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.590154 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.627350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:44.628523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] MFA Rules not processed for user `d9fc2512816d445594144a5bf55784ac`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:44.629929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:44.645590 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:44.654426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.656723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.673708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1297/2607] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:44 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:46:44.735741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1311/2608] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:44 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:44.749626 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.752923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:44.768588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36a6ff8-b81c-4e3c-b5c7-078add206a62 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.838797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.839854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.840384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.840907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.859382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:44.862827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:44.882157 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1298/2609] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:44 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:44.945913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:44.963191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512cf931-4866-4514-a66f-f7d9a4c4816f None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:44.970627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:44.971919 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:44.972455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:44.972926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:44.976382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:45.106133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:45.110123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:45.132074 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1312/2610] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:44 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:46.927653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:46.939413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fdd838-0793-4983-97ef-56ca6c2a1646 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:46.946940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:46.947614 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:46.947901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:46.948262 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:46.950155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:47.056787 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:47.066662 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4655abb5-db78-4e85-b324-6d1e15a739eb service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:47.075046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1299/2611] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:46 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:51.996217 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:52.115479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233e116-e64b-49e6-857b-420e91012b7c service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:52.121179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:52.121991 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:52.122166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:52.122337 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] PATH_INFO: `/v3/users/64d5f1c8de2e44c19962957dcfd9aebf` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:52.123928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=64d5f1c8de2e44c19962957dcfd9aebf)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:52.126594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:52.206279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64d5f1c8de2e44c19962957dcfd9aebf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:52.207867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64d5f1c8de2e44c19962957dcfd9aebf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:52.209167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64d5f1c8de2e44c19962957dcfd9aebf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:52.214783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:52.224814 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64d5f1c8de2e44c19962957dcfd9aebf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:52.240725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64d5f1c8de2e44c19962957dcfd9aebf'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:52.262682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1313/2612] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:51 2022] DELETE /identity/v3/users/64d5f1c8de2e44c19962957dcfd9aebf => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:46:52.267652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:52.268603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:52.268603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:52.291148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:52.293387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:52.381120 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1300/2613] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:52.391407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:52.393023 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:52.463728 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e463658-c188-49d2-ad63-85b02ec3c4f0 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:52.467830 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:52.468514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:52.468792 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:52.469555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:52.480787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:52.482245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:52.494647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1314/2614] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:46:52.938781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:52.940377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:52.956654 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff249d46-334a-473f-bd37-1ff383576e91 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:52.965616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:52.966502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:52.966897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:52.967319 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] PATH_INFO: `/v3/projects/06ffab0f61eb42f1b6f2cefe3d126a03` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:52.968786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06ffab0f61eb42f1b6f2cefe3d126a03)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:52.971060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:53.001714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:53.040854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06ffab0f61eb42f1b6f2cefe3d126a03', 'request_id': 'req-98524b2b-efff-4c25-8db0-29907b414279'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.051365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06ffab0f61eb42f1b6f2cefe3d126a03', 'request_id': 'req-98524b2b-efff-4c25-8db0-29907b414279'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.062522 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06ffab0f61eb42f1b6f2cefe3d126a03', 'request_id': 'req-98524b2b-efff-4c25-8db0-29907b414279'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.078119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] The token cache is being invalidate because project 06ffab0f61eb42f1b6f2cefe3d126a03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:53.078584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.079623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.082200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1301/2615] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:52 2022] DELETE /identity/v3/projects/06ffab0f61eb42f1b6f2cefe3d126a03 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:53.087303 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:53.106132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042cf294-455a-49b0-a826-e2c29b01635f service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:53.182129 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:53.182773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:53.183137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:53.183465 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] PATH_INFO: `/v3/users/e5f53ef1c4f943b99e12dbf1a104e7db` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:53.185126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5f53ef1c4f943b99e12dbf1a104e7db)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:53.187187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:53.245829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5f53ef1c4f943b99e12dbf1a104e7db'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.246604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5f53ef1c4f943b99e12dbf1a104e7db'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.247371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5f53ef1c4f943b99e12dbf1a104e7db'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.257726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5f53ef1c4f943b99e12dbf1a104e7db'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.271531 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5f53ef1c4f943b99e12dbf1a104e7db'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.282813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:53.303133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1315/2616] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:53 2022] DELETE /identity/v3/users/e5f53ef1c4f943b99e12dbf1a104e7db => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:53.625036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:53.627531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:53.650514 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98524b2b-efff-4c25-8db0-29907b414279 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:53.743993 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:53.744853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:53.745290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:53.745693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] PATH_INFO: `/v3/projects/1c8aad54ff534f8a86d76c5300eb6bbc` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:53.747075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c8aad54ff534f8a86d76c5300eb6bbc)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:53.749603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:53.827554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c8aad54ff534f8a86d76c5300eb6bbc', 'request_id': 'req-8942f16a-08b7-45bf-9491-b8cf165b0d8e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.837517 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c8aad54ff534f8a86d76c5300eb6bbc', 'request_id': 'req-8942f16a-08b7-45bf-9491-b8cf165b0d8e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.849330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c8aad54ff534f8a86d76c5300eb6bbc', 'request_id': 'req-8942f16a-08b7-45bf-9491-b8cf165b0d8e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.863246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] The token cache is being invalidate because project 1c8aad54ff534f8a86d76c5300eb6bbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:53.863623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.865084 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:53.866824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1302/2617] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:53 2022] DELETE /identity/v3/projects/1c8aad54ff534f8a86d76c5300eb6bbc => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:53.872042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:53.873689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:53.888535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1223d771-d8a0-4e27-95ae-db4b159b7f8b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:53.975863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:53.976845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:53.977300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:53.977729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] PATH_INFO: `/v3/users/d9fc2512816d445594144a5bf55784ac` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:53.979809 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9fc2512816d445594144a5bf55784ac)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:53.983594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:54.063141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9fc2512816d445594144a5bf55784ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.063999 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9fc2512816d445594144a5bf55784ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.065172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9fc2512816d445594144a5bf55784ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.075455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9fc2512816d445594144a5bf55784ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.090279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-14ce76e3-a7c1-46a4-a41d-ee4568b9821f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9fc2512816d445594144a5bf55784ac'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.114236 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1316/2618] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:46:53 2022] DELETE /identity/v3/users/d9fc2512816d445594144a5bf55784ac => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:54.460466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:54.462958 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:54.490552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8942f16a-08b7-45bf-9491-b8cf165b0d8e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:54.547440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:54.587479 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:54.588620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:54.589116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:54.589750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] PATH_INFO: `/v3/projects/9e054f77448b4211a88e1827774ec000` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:54.591822 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e054f77448b4211a88e1827774ec000)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:54.595640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:54.681105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e054f77448b4211a88e1827774ec000', 'request_id': 'req-637d92a3-4a42-436c-a173-47baa001dfc1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.690085 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e054f77448b4211a88e1827774ec000', 'request_id': 'req-637d92a3-4a42-436c-a173-47baa001dfc1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.699981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e054f77448b4211a88e1827774ec000', 'request_id': 'req-637d92a3-4a42-436c-a173-47baa001dfc1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.713738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] The token cache is being invalidate because project 9e054f77448b4211a88e1827774ec000 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:46:54.714033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.714736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-637d92a3-4a42-436c-a173-47baa001dfc1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:46:54.716314 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1303/2619] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:46:54 2022] DELETE /identity/v3/projects/9e054f77448b4211a88e1827774ec000 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:54.723317 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:54.723604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:54.723912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:54.745257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:54.746421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:54.834702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1317/2620] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:46:54.849625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:54.850304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:54.850670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:54.880094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:54.881645 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:54.891860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1304/2621] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:54.897196 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:54.913316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f53d9f9-f8a1-472e-979b-8a4fa16e51b2 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:54.918955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:54.919575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:54.919845 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:54.920151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:54.921009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:46:54.921452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:54.923273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:54.926365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:54.939954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1318/2622] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:46:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:46:54.944518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:54.955783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ee43a-a877-49db-92d8-c5a3cc6cc4cd None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:54.962045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:54.962910 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:54.963229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:54.963838 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:54.964575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:54.966789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:55.007078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1305/2623] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:54 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:55.012111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:55.036921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc64d93-eb16-424a-9aa0-6c07640e778e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:55.134959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:55.135787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:55.136013 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:55.136293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:55.137128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:55.139828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:55.175484 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1319/2624] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:55 2022] POST /identity/v3/users => generated 287 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:55.181605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:55.212270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf099656-97f4-4f59-8ea7-4c6be7ac4a84 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:55.220581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:55.221803 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:55.222462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:55.222946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:55.224154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:55.227263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:55.241185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1306/2625] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:55 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:55.245477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:55.257520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddea0d9-aeb0-4b7c-b6a8-b5e180461b62 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:55.263863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d804af-13f6-4b2e-b1c4-6af373d95e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:55.264912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d804af-13f6-4b2e-b1c4-6af373d95e07 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:55.265362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d804af-13f6-4b2e-b1c4-6af373d95e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:55.265962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d804af-13f6-4b2e-b1c4-6af373d95e07 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:55.267071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d804af-13f6-4b2e-b1c4-6af373d95e07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:55.270065 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d804af-13f6-4b2e-b1c4-6af373d95e07 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:55.292719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1320/2626] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:55 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:46:55.299379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:55.315202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974b19a3-7a95-453d-a356-d07a60deeb6e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:55.323145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:55.324377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:55.324908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:55.325231 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] PATH_INFO: `/v3/projects/95a0e41b3704462aab04efdf5ddeeda0/users/5121df04bea94af8a780dc64f5d04ca0/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:55.353199 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95a0e41b3704462aab04efdf5ddeeda0, user_id=5121df04bea94af8a780dc64f5d04ca0, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:55.358310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:55.379784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1307/2627] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:55 2022] PUT /identity/v3/projects/95a0e41b3704462aab04efdf5ddeeda0/users/5121df04bea94af8a780dc64f5d04ca0/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:55.387917 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6a477-aad1-4e15-b288-c6ff08936376 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:55.388445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6a477-aad1-4e15-b288-c6ff08936376 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:55.388737 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6a477-aad1-4e15-b288-c6ff08936376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:55.423532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-61a6a477-aad1-4e15-b288-c6ff08936376 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:55.425281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-61a6a477-aad1-4e15-b288-c6ff08936376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:55.534016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1321/2628] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:55.547121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:55.549322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:55.652253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbeee73-0fa5-4b76-8781-a3bea03c56a3 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:55.656382 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:55.657333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:55.657659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:55.657995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:55.668146 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:55.669700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:55.681025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1308/2629] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:46:56.553591 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:56.554495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:56.554710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:56.603256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] MFA Rules not processed for user `5121df04bea94af8a780dc64f5d04ca0`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:56.604559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:56.759390 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1322/2630] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:56.765455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:56.778652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee1673b-7ab0-421e-ac6f-84cb8af73482 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:56.786158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:56.786952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:56.787353 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:56.787809 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:56.788650 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:56.791791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:56.805397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:56.848008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1309/2631] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:46:56 2022] POST /identity/v3/projects => generated 371 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:46:56.852918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:56.860723 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:46:56.869698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e6b321-1391-4651-8e99-79abd8edd478 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:56.965103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:56.965905 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:56.967024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:56.967443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:56.968224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:56.970906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:57.009493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1323/2632] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:46:56 2022] POST /identity/v3/users => generated 292 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:46:57.017286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:57.034089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1aa0ff7-a34b-4638-95b8-fe432de03a1d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:57.039724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:57.040509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:57.040834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:57.041185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:57.041887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:57.043554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:57.056485 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1310/2633] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:57 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:57.061179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:57.072861 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8c222-f3d5-4cd7-ac21-ab75acd4231a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:57.078277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:57.078971 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:57.079263 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:57.079543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] PATH_INFO: `/v3/projects/430cb741f8fe4b3c983937bd44782a20/users/9219a32415ec4dd8a9056275bd7a63c4/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:57.097910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] RBAC: Authorizing `identity:create_grant(project_id=430cb741f8fe4b3c983937bd44782a20, user_id=9219a32415ec4dd8a9056275bd7a63c4, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:57.100640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:57.118245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1324/2634] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:57 2022] PUT /identity/v3/projects/430cb741f8fe4b3c983937bd44782a20/users/9219a32415ec4dd8a9056275bd7a63c4/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:57.123307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:57.142657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d28a845-8b19-4b2e-a0e1-1a5a8e9f199b admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:57.257416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683877cd-8a86-45f4-add0-3fb40d882506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:57.258179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683877cd-8a86-45f4-add0-3fb40d882506 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:57.258559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683877cd-8a86-45f4-add0-3fb40d882506 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:57.258951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683877cd-8a86-45f4-add0-3fb40d882506 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:57.259744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683877cd-8a86-45f4-add0-3fb40d882506 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:57.261760 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683877cd-8a86-45f4-add0-3fb40d882506 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:57.273976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1311/2635] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:46:57 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:46:57.280682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:57.300338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09fb195-c98a-4624-9c66-8beb4fe0797a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:57.308450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:57.309566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:57.310059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:57.310558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] PATH_INFO: `/v3/projects/430cb741f8fe4b3c983937bd44782a20/users/9219a32415ec4dd8a9056275bd7a63c4/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:57.314684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=430cb741f8fe4b3c983937bd44782a20, user_id=9219a32415ec4dd8a9056275bd7a63c4, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:57.319986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:57.340040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1325/2636] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:46:57 2022] PUT /identity/v3/projects/430cb741f8fe4b3c983937bd44782a20/users/9219a32415ec4dd8a9056275bd7a63c4/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:46:58.161165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:58.162104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:58.162286 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:58.205872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] MFA Rules not processed for user `9219a32415ec4dd8a9056275bd7a63c4`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:46:58.207741 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:58.323173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1312/2637] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:46:58 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:46:58.344763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:46:58.353233 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:46:58.450451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15f71b8-c6db-47c9-a558-c64a001919c3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:46:58.454564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:46:58.455184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:46:58.455462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:46:58.455734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:46:58.536411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:46:58.537918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:46:58.549016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1326/2638] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:46:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:00.146801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:00.148529 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:00.164205 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554b9e6f-8791-44c6-96d1-7b5e01358dd7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:00.173769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2c5e81-e7f9-4966-a460-1627fbf03973 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:00.174419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2c5e81-e7f9-4966-a460-1627fbf03973 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:00.174842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2c5e81-e7f9-4966-a460-1627fbf03973 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:00.175175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2c5e81-e7f9-4966-a460-1627fbf03973 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:00.195409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2c5e81-e7f9-4966-a460-1627fbf03973 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:00.197118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2c5e81-e7f9-4966-a460-1627fbf03973 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:00.211206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1313/2639] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:00 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:01.183149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:01.185391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:01.208394 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:01.277582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e7bc95-9b85-480c-836e-4f608afda99a service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:01.282805 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:01.283472 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:01.283759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:01.284036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] PATH_INFO: `/v3/users/ba24df9dc4c94ccd96d0d977a3aa529f` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:01.285302 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba24df9dc4c94ccd96d0d977a3aa529f)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:01.287273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:01.352428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba24df9dc4c94ccd96d0d977a3aa529f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:01.353138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba24df9dc4c94ccd96d0d977a3aa529f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:01.353794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba24df9dc4c94ccd96d0d977a3aa529f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:01.363539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba24df9dc4c94ccd96d0d977a3aa529f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:01.371954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba24df9dc4c94ccd96d0d977a3aa529f'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:01.395093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1327/2640] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:01 2022] DELETE /identity/v3/users/ba24df9dc4c94ccd96d0d977a3aa529f => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:01.401094 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:01.402227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:01.402227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:01.432667 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:01.434035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:01.480853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:01.507862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1314/2641] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:01.520180 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:01.522311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:01.613038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4a4597-e091-4cad-9ba9-e48dade86b48 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:01.617828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:01.618744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:01.619185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:01.619635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:01.632397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:01.634080 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:01.650678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1328/2642] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:02.209493 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:02.212105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:02.232884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed850cb2-93ac-4af2-a2ad-cfcb775807d2 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:02.240300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:02.241767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:02.242396 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:02.243023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] PATH_INFO: `/v3/projects/fb68c2d1eb3c437798cbb2828591cf02` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:02.245537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb68c2d1eb3c437798cbb2828591cf02)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:02.249395 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:02.335377 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb68c2d1eb3c437798cbb2828591cf02', 'request_id': 'req-02cd69e6-c852-455f-9e36-b9a9dd4c92de'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.346161 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb68c2d1eb3c437798cbb2828591cf02', 'request_id': 'req-02cd69e6-c852-455f-9e36-b9a9dd4c92de'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.362797 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb68c2d1eb3c437798cbb2828591cf02', 'request_id': 'req-02cd69e6-c852-455f-9e36-b9a9dd4c92de'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.382138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] The token cache is being invalidate because project fb68c2d1eb3c437798cbb2828591cf02 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:02.382471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.383246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.384888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1315/2643] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:02 2022] DELETE /identity/v3/projects/fb68c2d1eb3c437798cbb2828591cf02 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:02.389947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:02.404464 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3888eb4-966c-47cf-88e9-c6272f9bf6fd service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:02.454026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:02.488621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:02.489502 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:02.489955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:02.490375 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] PATH_INFO: `/v3/users/211a8dc8ada44fbe9a3155ba651c7008` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:02.492215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] RBAC: Authorizing `identity:delete_user(user_id=211a8dc8ada44fbe9a3155ba651c7008)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:02.495301 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:02.561380 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '211a8dc8ada44fbe9a3155ba651c7008'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.562407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '211a8dc8ada44fbe9a3155ba651c7008'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.563765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '211a8dc8ada44fbe9a3155ba651c7008'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.573763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '211a8dc8ada44fbe9a3155ba651c7008'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.582767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '211a8dc8ada44fbe9a3155ba651c7008'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:02.611685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1329/2644] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:02 2022] DELETE /identity/v3/users/211a8dc8ada44fbe9a3155ba651c7008 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:02.980976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:02.983018 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:03.001112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cd69e6-c852-455f-9e36-b9a9dd4c92de admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:03.077017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:03.077963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:03.078394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:03.078811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] PATH_INFO: `/v3/projects/142e17cb5a164d5cb7423b813d49b30c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:03.080679 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] RBAC: Authorizing `identity:delete_project(project_id=142e17cb5a164d5cb7423b813d49b30c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:03.083995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:03.178859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '142e17cb5a164d5cb7423b813d49b30c', 'request_id': 'req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:03.189651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '142e17cb5a164d5cb7423b813d49b30c', 'request_id': 'req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:03.208593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '142e17cb5a164d5cb7423b813d49b30c', 'request_id': 'req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:03.212840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:03.243965 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] The token cache is being invalidate because project 142e17cb5a164d5cb7423b813d49b30c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:03.245036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:03.249047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-5b5bb8e4-6fb3-4e8c-8902-d4558cc9e586 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:03.253039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1316/2645] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:02 2022] DELETE /identity/v3/projects/142e17cb5a164d5cb7423b813d49b30c => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:05.494623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:05.496568 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:05.511213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d9ea7-27a7-400a-be2e-0e47a3b3b325 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:05.578942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:05.579494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:05.579775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:05.580053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] PATH_INFO: `/v3/users/ef0027d9b42b495f8a0e4c546c809b3d` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:05.581361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef0027d9b42b495f8a0e4c546c809b3d)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:05.583360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:05.650523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef0027d9b42b495f8a0e4c546c809b3d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:05.651613 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef0027d9b42b495f8a0e4c546c809b3d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:05.652856 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef0027d9b42b495f8a0e4c546c809b3d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:05.666617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef0027d9b42b495f8a0e4c546c809b3d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:05.673250 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:05.682318 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef0027d9b42b495f8a0e4c546c809b3d'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:05.708963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1330/2646] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:05 2022] DELETE /identity/v3/users/ef0027d9b42b495f8a0e4c546c809b3d => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:05.715105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:05.715725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:05.715725 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:05.739379 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:05.740716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:05.820224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1317/2647] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:05.833437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:05.835241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:05.908169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f3ef5-8a30-4e44-9770-1543ecb6a514 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:05.912117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:05.912791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:05.913076 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:05.913353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:05.924391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:05.925863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:05.937017 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1331/2648] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:06.324867 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:06.327726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:06.352370 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d59763-bfbd-4b19-9cd4-b1769a7570b0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:06.358246 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:06.359842 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:06.360200 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:06.360816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] PATH_INFO: `/v3/projects/fb107a3b551d4161a6eef6640ec23606` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:06.362611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb107a3b551d4161a6eef6640ec23606)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:06.364792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:06.448635 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb107a3b551d4161a6eef6640ec23606', 'request_id': 'req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:06.460184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb107a3b551d4161a6eef6640ec23606', 'request_id': 'req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:06.473530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb107a3b551d4161a6eef6640ec23606', 'request_id': 'req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:06.487922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] The token cache is being invalidate because project fb107a3b551d4161a6eef6640ec23606 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:06.488408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:06.489450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-cbe6e9e2-1010-4ced-91b9-a763eb4f4540 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:06.491304 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1318/2649] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:06 2022] DELETE /identity/v3/projects/fb107a3b551d4161a6eef6640ec23606 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:11.577236 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:11.581467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:11.595312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f860645-d97a-4c4d-8682-5ce7a51e16a7 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:11.668124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:11.668932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:11.669097 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:11.669445 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] PATH_INFO: `/v3/users/a4046b5c5e08451e9f26e882b6c69893` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:11.670679 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4046b5c5e08451e9f26e882b6c69893)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:11.673223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:11.732967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4046b5c5e08451e9f26e882b6c69893'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:11.733678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4046b5c5e08451e9f26e882b6c69893'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:11.735006 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4046b5c5e08451e9f26e882b6c69893'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:11.744491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4046b5c5e08451e9f26e882b6c69893'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:11.754269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4046b5c5e08451e9f26e882b6c69893'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:11.766626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:11.778008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1332/2650] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:11 2022] DELETE /identity/v3/users/a4046b5c5e08451e9f26e882b6c69893 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:11.783961 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:11.784832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:11.784832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:11.812833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:11.814728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:11.900500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1319/2651] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:11.911474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:11.913668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:12.003486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6889b4-ea0d-4789-87cc-781458201fd9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.009082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ffbc64-a658-4f27-b25e-239099173e1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.009667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ffbc64-a658-4f27-b25e-239099173e1f service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.010019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ffbc64-a658-4f27-b25e-239099173e1f service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.010359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ffbc64-a658-4f27-b25e-239099173e1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.023061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ffbc64-a658-4f27-b25e-239099173e1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.023675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ffbc64-a658-4f27-b25e-239099173e1f service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.036085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1333/2652] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:12.404788 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.407474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:12.422550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bb7853-112c-45fd-9da5-8322524287b8 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.427612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.428410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.428753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.429105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] PATH_INFO: `/v3/projects/b3a88556db5848d287847eed9bf8b0e1` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.430442 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3a88556db5848d287847eed9bf8b0e1)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.432664 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.453595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:12.493866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3a88556db5848d287847eed9bf8b0e1', 'request_id': 'req-47b47f01-0a27-40b9-b46f-9b551d20af70'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:12.501814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3a88556db5848d287847eed9bf8b0e1', 'request_id': 'req-47b47f01-0a27-40b9-b46f-9b551d20af70'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:12.510544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3a88556db5848d287847eed9bf8b0e1', 'request_id': 'req-47b47f01-0a27-40b9-b46f-9b551d20af70'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:12.524661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] The token cache is being invalidate because project b3a88556db5848d287847eed9bf8b0e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:12.525026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:12.525795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-47b47f01-0a27-40b9-b46f-9b551d20af70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:12.527427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1320/2653] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:12 2022] DELETE /identity/v3/projects/b3a88556db5848d287847eed9bf8b0e1 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:12.533459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.533744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.534052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.556988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:12.558297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:12.628773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1334/2654] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:12.638043 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.638358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.638640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.660494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:12.661749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:12.669999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1321/2655] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:12.674487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.684802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146cbbe6-606e-452b-8ccd-4f04af8bf627 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.690655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.691288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.691572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.691849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.692569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:12.692873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.694522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.708132 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1335/2656] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:47:12.713186 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.723097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2187bd2c-1b7a-4ab1-8c58-6ca76390e433 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.728452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.729096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.729373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.729638 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.730263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.732356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.764343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1322/2657] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:12 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:12.769428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.780551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee2d837-89ca-4189-aa5a-4ebbad7bbc0f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.840536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.841158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.841442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.841720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.842367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.844397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.867826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1336/2658] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:12 2022] POST /identity/v3/users => generated 288 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:47:12.872531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.884002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0293845a-ad69-416e-9a1f-053743ac9567 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.888954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.889589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.889872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.890147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.891317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.892989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.904208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1323/2659] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:12.909268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.920387 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5cf3b-c778-4110-b509-f115a2128b22 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.925684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5bc99b-68d5-41fa-a22c-9386759a4b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.926336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5bc99b-68d5-41fa-a22c-9386759a4b88 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.926619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5bc99b-68d5-41fa-a22c-9386759a4b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.926896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5bc99b-68d5-41fa-a22c-9386759a4b88 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.927550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5bc99b-68d5-41fa-a22c-9386759a4b88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.929328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5bc99b-68d5-41fa-a22c-9386759a4b88 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.941265 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1337/2660] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:12.946394 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:12.958167 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfc89d-967c-47ad-8ab7-ad0b23a7f677 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:12.963190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:12.963864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:12.964142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:12.964486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] PATH_INFO: `/v3/projects/ed8daf335dbd45219e0d7558c2cdfed9/users/13e21c17d32a418a84943e53c6185984/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:12.980972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed8daf335dbd45219e0d7558c2cdfed9, user_id=13e21c17d32a418a84943e53c6185984, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:12.983644 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:12.998371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1324/2661] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:12 2022] PUT /identity/v3/projects/ed8daf335dbd45219e0d7558c2cdfed9/users/13e21c17d32a418a84943e53c6185984/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:13.010010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:13.010420 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:13.010720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:13.018974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=1,keystone:SELECT=96,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:13.045051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] MFA Rules not processed for user `13e21c17d32a418a84943e53c6185984`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:13.046282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:13.120670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1338/2662] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:13.132572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:13.134951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:13.209462 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d84e6b-12d7-4e94-9acc-79aabe6ca61a admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:13.213592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:13.214223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:13.214508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:13.214784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:13.225113 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:13.226601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:13.236689 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1325/2663] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:14.770232 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:14.773053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:14.862998 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75082fa3-f055-4120-afbf-c74f2987d68b None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:14.868071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:14.868873 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:14.869223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:14.869570 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] PATH_INFO: `/v3/users/5121df04bea94af8a780dc64f5d04ca0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:14.870945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5121df04bea94af8a780dc64f5d04ca0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:14.873158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:14.935947 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5121df04bea94af8a780dc64f5d04ca0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:14.936667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5121df04bea94af8a780dc64f5d04ca0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:14.938094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5121df04bea94af8a780dc64f5d04ca0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:14.950778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5121df04bea94af8a780dc64f5d04ca0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:14.960854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5121df04bea94af8a780dc64f5d04ca0'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:14.985176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1339/2664] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:14 2022] DELETE /identity/v3/users/5121df04bea94af8a780dc64f5d04ca0 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:15.232065 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:15.254441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a4a757-7d4a-4b5f-be8a-a5a6c7277949 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:15.341531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:15.370856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:15.371503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:15.371786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:15.372056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] PATH_INFO: `/v3/projects/95a0e41b3704462aab04efdf5ddeeda0` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:15.373409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95a0e41b3704462aab04efdf5ddeeda0)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:15.375945 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:15.450263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95a0e41b3704462aab04efdf5ddeeda0', 'request_id': 'req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.460722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95a0e41b3704462aab04efdf5ddeeda0', 'request_id': 'req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.473743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95a0e41b3704462aab04efdf5ddeeda0', 'request_id': 'req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.499436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] The token cache is being invalidate because project 95a0e41b3704462aab04efdf5ddeeda0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:15.500653 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.501500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.503578 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1326/2665] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:15 2022] DELETE /identity/v3/projects/95a0e41b3704462aab04efdf5ddeeda0 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:15.508785 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:15.510663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:15.529422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4057844e-06c7-483a-9c1f-9e06937fae5f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:15.540876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:15.608501 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:15.609300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:15.609564 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:15.609841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] PATH_INFO: `/v3/users/9219a32415ec4dd8a9056275bd7a63c4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:15.611096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9219a32415ec4dd8a9056275bd7a63c4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:15.613138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:15.671181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9219a32415ec4dd8a9056275bd7a63c4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.671847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9219a32415ec4dd8a9056275bd7a63c4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.672511 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9219a32415ec4dd8a9056275bd7a63c4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.682113 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9219a32415ec4dd8a9056275bd7a63c4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.693696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8de87f3f-4376-4459-b1f1-ed6f67343df7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9219a32415ec4dd8a9056275bd7a63c4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:15.717656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1340/2666] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:15 2022] DELETE /identity/v3/users/9219a32415ec4dd8a9056275bd7a63c4 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:15.936444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:15.938946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:15.962126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8238b5-2f13-41dc-93f4-83b4db6b5a06 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.032774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.033436 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.033718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.033998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] PATH_INFO: `/v3/projects/430cb741f8fe4b3c983937bd44782a20` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.035194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=430cb741f8fe4b3c983937bd44782a20)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.037207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.096369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '430cb741f8fe4b3c983937bd44782a20', 'request_id': 'req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.105031 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '430cb741f8fe4b3c983937bd44782a20', 'request_id': 'req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.114815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '430cb741f8fe4b3c983937bd44782a20', 'request_id': 'req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.131982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] The token cache is being invalidate because project 430cb741f8fe4b3c983937bd44782a20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:16.132403 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.133136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-656e61e5-2d69-47d9-861b-c7a7bd0d0ce1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.134865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1327/2667] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:15 2022] DELETE /identity/v3/projects/430cb741f8fe4b3c983937bd44782a20 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:16.141646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.142513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.142513 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.165697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:16.167240 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.238443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1341/2668] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:16.246983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.247659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.247659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.256677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:16.272029 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:16.274187 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.283499 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1328/2669] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:16.287913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.299014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b847d2-83eb-43e2-93a9-7cbe0d2a2a12 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.306243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.307184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.307684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.308381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.309597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:16.310292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.313136 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.327599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1342/2670] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:47:16.332729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.344711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c105fe76-9b23-44df-90e4-aaffbaa298e5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.350210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.350835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.351115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.351534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.352175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.359174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.385657 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1329/2671] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:16 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:16.390225 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.401308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78879482-b30a-477f-b08f-62acb12a46a7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.415899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:16.482346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.482974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.483259 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.483536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.484212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.487146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.510119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1343/2672] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:16 2022] POST /identity/v3/users => generated 292 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:47:16.516580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.528184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a570bd52-45e3-4bb4-bb0f-0097469b45b9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.535426 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.536214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.536729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.536899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.537768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.539469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.552537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1330/2673] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:16.557258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.568966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1517534-cf78-42ca-b956-544a40ab10ea admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.574770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.575620 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.576049 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.576319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.577039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.578762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.592219 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1344/2674] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:16.594470 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.596339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.597888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.610781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec66fcd9-0757-4373-ab79-0461602dbf47 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.611274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08b39e2-7421-4ebd-b205-26778206bfd4 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.615855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a5fb46-2e1b-4755-a9a9-1426225d6463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.616528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.616758 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5fb46-2e1b-4755-a9a9-1426225d6463 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.617117 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5fb46-2e1b-4755-a9a9-1426225d6463 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.617307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.617484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.617614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a5fb46-2e1b-4755-a9a9-1426225d6463 admin admin] PATH_INFO: `/v3/projects/e5748dfaec3a44739b34d2ccb11f01aa/users/b95291918b7a45c488460c0a2e134120/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.617810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] PATH_INFO: `/v3/users/13e21c17d32a418a84943e53c6185984` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.619069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=13e21c17d32a418a84943e53c6185984)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.621092 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.638071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a5fb46-2e1b-4755-a9a9-1426225d6463 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5748dfaec3a44739b34d2ccb11f01aa, user_id=b95291918b7a45c488460c0a2e134120, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.641779 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a5fb46-2e1b-4755-a9a9-1426225d6463 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.661029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1345/2675] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:16 2022] PUT /identity/v3/projects/e5748dfaec3a44739b34d2ccb11f01aa/users/b95291918b7a45c488460c0a2e134120/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:16.672578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.673000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.673456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.679885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13e21c17d32a418a84943e53c6185984'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.689824 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13e21c17d32a418a84943e53c6185984'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.698789 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13e21c17d32a418a84943e53c6185984'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.699563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13e21c17d32a418a84943e53c6185984'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.700943 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7cab98f2-6f4b-4550-8aad-34457f234b6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13e21c17d32a418a84943e53c6185984'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:16.708426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] MFA Rules not processed for user `b95291918b7a45c488460c0a2e134120`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:16.709701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.726292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1331/2676] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:16 2022] DELETE /identity/v3/users/13e21c17d32a418a84943e53c6185984 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:16.731032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.731320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.731762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.753621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:16.754896 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.789539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1346/2677] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:16.802210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.803734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.827299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1332/2678] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:16.841295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:16.842852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:16.882397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4bfff4-c3cb-45c4-94a7-7180956b85b9 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.894505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:16.899432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:16.921895 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d06025-e550-4879-84c7-d98d2ca77392 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:16.927047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.927915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.928315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.928752 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.939860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:16.941417 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:16.946631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360ae215-5efd-44a0-981a-734a659ea8e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:16.947270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360ae215-5efd-44a0-981a-734a659ea8e1 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:16.947549 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360ae215-5efd-44a0-981a-734a659ea8e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:16.947827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360ae215-5efd-44a0-981a-734a659ea8e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:16.956536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1333/2679] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:17.023799 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360ae215-5efd-44a0-981a-734a659ea8e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.025929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360ae215-5efd-44a0-981a-734a659ea8e1 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.040512 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1347/2680] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:17.074116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.075526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.091446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57ec4d6-ccae-40b2-a9f5-b1aa7e07988d service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.098071 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.099640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.099640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.099640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] PATH_INFO: `/v3/users/b95291918b7a45c488460c0a2e134120` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.101083 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b95291918b7a45c488460c0a2e134120)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.103983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.175035 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b95291918b7a45c488460c0a2e134120'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.184565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b95291918b7a45c488460c0a2e134120'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b95291918b7a45c488460c0a2e134120'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.196498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b95291918b7a45c488460c0a2e134120'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.197140 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b95291918b7a45c488460c0a2e134120'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.223775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1334/2681] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:17 2022] DELETE /identity/v3/users/b95291918b7a45c488460c0a2e134120 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:17.229191 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.229755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.229755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.251038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:17.252552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.326887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.332392 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.350053 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1348/2682] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:17.352005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123b3cf3-e692-4663-8753-cb414ea39089 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.358430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.360648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.361730 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.362249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.362367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] PATH_INFO: `/v3/projects/ed8daf335dbd45219e0d7558c2cdfed9` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.363661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.364429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed8daf335dbd45219e0d7558c2cdfed9)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.368952 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.464363 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51ae1fa-c4f9-48c8-af27-c9ec89952d3d None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.464878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed8daf335dbd45219e0d7558c2cdfed9', 'request_id': 'req-e5159094-ed81-4f2d-93ab-a0640535a940'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.473596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed8daf335dbd45219e0d7558c2cdfed9', 'request_id': 'req-e5159094-ed81-4f2d-93ab-a0640535a940'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.483768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed8daf335dbd45219e0d7558c2cdfed9', 'request_id': 'req-e5159094-ed81-4f2d-93ab-a0640535a940'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.497508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] The token cache is being invalidate because project ed8daf335dbd45219e0d7558c2cdfed9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:17.497807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.498503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e5159094-ed81-4f2d-93ab-a0640535a940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:17.500728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1335/2683] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:17 2022] DELETE /identity/v3/projects/ed8daf335dbd45219e0d7558c2cdfed9 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:17.505776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dd98e8-ac32-4b15-84d8-6b1b6c2f6731 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.506141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dd98e8-ac32-4b15-84d8-6b1b6c2f6731 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.506397 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=100 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:17.506563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dd98e8-ac32-4b15-84d8-6b1b6c2f6731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.529778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-c1dd98e8-ac32-4b15-84d8-6b1b6c2f6731 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:17.532115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-c1dd98e8-ac32-4b15-84d8-6b1b6c2f6731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.538406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.539532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.540094 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.540678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.543368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.568786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:17.609050 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1336/2684] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:17.619694 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.620066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.620378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.625862 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.627410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.638414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1349/2685] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:17.643779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:17.645090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:17.653454 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1337/2686] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:17.658668 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.670438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3478d55-5101-446e-b0e4-6afe9d12a358 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.675772 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.677197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.677575 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.677903 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.678532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:17.678833 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.680545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.694151 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1350/2687] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:47:17.701575 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.714630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b3aee9-18a4-4bb9-a02d-676801ccfac1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.721983 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.723074 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.723685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.724360 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.725242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.728543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.772715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1338/2688] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:17 2022] POST /identity/v3/projects => generated 369 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:17.773968 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.786086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670b4989-2dec-4d75-aaf7-8029763308c1 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.862199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.862973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.863349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.863702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.864474 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.866732 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.900753 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1351/2689] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:17 2022] POST /identity/v3/users => generated 285 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:17.903208 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.915323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a121ed-5c4a-49c0-b40f-8dfde52737b5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.920814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.921438 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.922437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.922859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.923606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.925300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.937152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1339/2690] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:17 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:17.942273 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:17.953709 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5122a989-d1ba-4df0-9545-61e5e9fefc36 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:17.959543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:17.960181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:17.960495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:17.960762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:17.961428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:17.963131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:17.978514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1352/2691] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:17 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:17.983811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.003106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.003569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b14d0a-0785-40cc-a47f-0e8bcd8a2e1e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.006091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.010525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1c0ff3-5f2a-4afa-8e6d-4a52e4fc3d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.011401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1c0ff3-5f2a-4afa-8e6d-4a52e4fc3d6e admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.011778 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1c0ff3-5f2a-4afa-8e6d-4a52e4fc3d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.012075 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1c0ff3-5f2a-4afa-8e6d-4a52e4fc3d6e admin admin] PATH_INFO: `/v3/projects/303ece81a3e84ae6b979c9bce3847c72/users/41882a2a38f342cb8badc204fc161108/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.026115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39d95df-d4ac-45b2-9fb6-a197a651d0a9 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.031670 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.032781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.033275 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.033696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] PATH_INFO: `/v3/projects/e5748dfaec3a44739b34d2ccb11f01aa` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.035411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5748dfaec3a44739b34d2ccb11f01aa)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.038588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.056550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1c0ff3-5f2a-4afa-8e6d-4a52e4fc3d6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=303ece81a3e84ae6b979c9bce3847c72, user_id=41882a2a38f342cb8badc204fc161108, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.061711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1c0ff3-5f2a-4afa-8e6d-4a52e4fc3d6e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.080330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1340/2692] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:17 2022] PUT /identity/v3/projects/303ece81a3e84ae6b979c9bce3847c72/users/41882a2a38f342cb8badc204fc161108/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:18.092153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.093021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.093535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.096635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:18.105764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5748dfaec3a44739b34d2ccb11f01aa', 'request_id': 'req-290ce239-6d11-49f0-8f32-f85f9848cba8'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:18.115962 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5748dfaec3a44739b34d2ccb11f01aa', 'request_id': 'req-290ce239-6d11-49f0-8f32-f85f9848cba8'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:18.127844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5748dfaec3a44739b34d2ccb11f01aa', 'request_id': 'req-290ce239-6d11-49f0-8f32-f85f9848cba8'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:18.136944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] MFA Rules not processed for user `41882a2a38f342cb8badc204fc161108`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:18.138994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.147934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] The token cache is being invalidate because project e5748dfaec3a44739b34d2ccb11f01aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:18.148362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:18.149508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-290ce239-6d11-49f0-8f32-f85f9848cba8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:18.151891 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1353/2693] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:18 2022] DELETE /identity/v3/projects/e5748dfaec3a44739b34d2ccb11f01aa => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:18.157115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc0cac-4370-4083-8737-e8a9c350545f None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.157545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc0cac-4370-4083-8737-e8a9c350545f None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.157959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc0cac-4370-4083-8737-e8a9c350545f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.181635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-b1bc0cac-4370-4083-8737-e8a9c350545f None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:18.182895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b1bc0cac-4370-4083-8737-e8a9c350545f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.230853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1341/2694] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:18.243775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.245274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.257701 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1354/2695] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:18.267749 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.268038 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.268352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.290087 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:18.291374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.300276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1355/2696] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:18.304393 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.314410 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49a4f03-9af7-4160-aff0-9f769f538037 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.319614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.320409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.320693 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.320963 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.321775 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:18.322110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.324081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.327335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150a89ac-5dcb-4495-bebb-5d8c9532f5e1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.332257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.333142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.333489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.333812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.338216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1356/2697] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:47:18.345223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.347980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.349613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.358029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3105e03b-a68d-48fb-b364-3d444ddacf88 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.359786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1342/2698] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:18 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:18.363245 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.363875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.364157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.364460 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.365100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.367081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.407000 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1357/2699] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:18 2022] POST /identity/v3/projects => generated 377 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:47:18.408209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.413883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:18.421682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdc039-13e0-4b5a-89f5-9c9e3b6251ad service glance] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.513616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.514560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.515044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.516759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.520826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.558410 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1343/2700] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:18 2022] POST /identity/v3/users => generated 289 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:18.563181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.574878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47740833-b4f6-4806-a044-a60c22eae05f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.579898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.580552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.580835 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.581111 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.581769 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.583449 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.596627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1358/2701] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:18.601354 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.613962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20221adb-a889-4cd7-8d31-d8078a2e6f08 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.619361 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab6229d-33da-4214-8c79-97fd80c02ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.620011 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab6229d-33da-4214-8c79-97fd80c02ce3 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.620330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab6229d-33da-4214-8c79-97fd80c02ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.620612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab6229d-33da-4214-8c79-97fd80c02ce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.621281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab6229d-33da-4214-8c79-97fd80c02ce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.622899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab6229d-33da-4214-8c79-97fd80c02ce3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.635139 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1344/2702] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:18.639747 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.651666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8e0164-1072-41ad-aa5c-044cffbb3751 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.657921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.658681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.659086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.659467 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] PATH_INFO: `/v3/projects/8be5e93b1f2a4ba5b016fe0136a3b8a7/users/dd8fc8ea008f4b8a9aef38f0f3d075bc/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.679579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8be5e93b1f2a4ba5b016fe0136a3b8a7, user_id=dd8fc8ea008f4b8a9aef38f0f3d075bc, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:18.684070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:18.706334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1359/2703] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:18 2022] PUT /identity/v3/projects/8be5e93b1f2a4ba5b016fe0136a3b8a7/users/dd8fc8ea008f4b8a9aef38f0f3d075bc/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:18.717705 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.718076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.718992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.758878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] MFA Rules not processed for user `dd8fc8ea008f4b8a9aef38f0f3d075bc`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:18.760706 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.848720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1345/2704] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:18.873176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:18.875052 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:18.954945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f3989f-7bdb-402a-8608-895d6ca7a901 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:18.963306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:18.965202 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:18.966146 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:18.966874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:18.993699 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:19.001005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:19.002914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:19.017755 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1360/2705] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:25.328857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:25.330549 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:25.427724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a9470e-0866-4c72-a360-4caf5815ada5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:25.434600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:25.435299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:25.435631 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:25.435957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:25.437980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:25.472969 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:25.526891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:25.528407 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:25.542399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1346/2706] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:29.026980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:29.987824 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:29.989610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:30.007079 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cde2fc-c09e-42d6-92ce-95b15f93cd8f service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:30.012145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:30.012849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:30.013133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:30.013406 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] PATH_INFO: `/v3/users/e65f0f1ba54f483abff788974b1256cb` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:30.014691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e65f0f1ba54f483abff788974b1256cb)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:30.016764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:30.083212 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e65f0f1ba54f483abff788974b1256cb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.084438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e65f0f1ba54f483abff788974b1256cb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.085226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e65f0f1ba54f483abff788974b1256cb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.095177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e65f0f1ba54f483abff788974b1256cb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.104569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-0e3cd9a5-5cdf-4a7c-bd89-db499dba6c2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e65f0f1ba54f483abff788974b1256cb'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.127889 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1361/2707] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:29 2022] DELETE /identity/v3/users/e65f0f1ba54f483abff788974b1256cb => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:30.409288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:30.424956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875c522d-33e9-48a0-9439-f12efb8a0161 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:30.529290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:30.529936 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:30.530322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:30.530620 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] PATH_INFO: `/v3/projects/35820b4f79774ac49f43fce915f56d6f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:30.533572 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35820b4f79774ac49f43fce915f56d6f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:30.535981 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:30.635734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35820b4f79774ac49f43fce915f56d6f', 'request_id': 'req-074e4195-65d5-4b78-b981-0f0cb43e63c2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.647333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35820b4f79774ac49f43fce915f56d6f', 'request_id': 'req-074e4195-65d5-4b78-b981-0f0cb43e63c2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.659810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35820b4f79774ac49f43fce915f56d6f', 'request_id': 'req-074e4195-65d5-4b78-b981-0f0cb43e63c2'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.680421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] The token cache is being invalidate because project 35820b4f79774ac49f43fce915f56d6f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:30.680908 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.682430 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-074e4195-65d5-4b78-b981-0f0cb43e63c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:30.685249 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1347/2708] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:30 2022] DELETE /identity/v3/projects/35820b4f79774ac49f43fce915f56d6f => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:30.694934 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:30.695343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:30.695803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:30.725636 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:30.726908 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:30.811450 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1362/2709] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:30.821212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:30.821508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:30.821946 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:30.855273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:30.857352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:30.871600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1348/2710] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:30.877717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:30.894439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5506ed-7681-4d71-ae98-a7061d8c130a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:30.900573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:30.901285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:30.901577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:30.901847 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:30.902515 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:30.902844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:30.904519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:30.917918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1363/2711] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:47:30.923608 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:30.941169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8a5ac7-5272-4cb3-b134-13d3227c3b93 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:30.949284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:30.950318 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:30.950817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:30.951322 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:30.952510 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:30.957541 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:31.009206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1349/2712] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:30 2022] POST /identity/v3/projects => generated 375 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:47:31.014256 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:31.029523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1fc7fd-abcb-4e76-9594-6a0bf1f960e4 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:31.116352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:31.117396 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:31.117880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:31.118360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:31.119523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:31.123298 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:31.136592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:31.163757 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1364/2713] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:31 2022] POST /identity/v3/users => generated 288 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:31.165183 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:31.176849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f8dc32-cdd3-4814-a20c-78b5d12b0254 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:31.182401 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:31.183032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:31.183357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:31.183599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:31.184331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:31.185990 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:31.197648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1350/2714] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:47:31.202731 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:31.220185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9164f51b-a438-4f1f-9513-376d30b862e2 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:31.228837 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a239d602-a62d-4182-ae90-84e1e70f272d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:31.230004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a239d602-a62d-4182-ae90-84e1e70f272d admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:31.230519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a239d602-a62d-4182-ae90-84e1e70f272d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:31.230980 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a239d602-a62d-4182-ae90-84e1e70f272d admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:31.232189 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a239d602-a62d-4182-ae90-84e1e70f272d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:31.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a239d602-a62d-4182-ae90-84e1e70f272d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:31.256925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1365/2715] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:31.262735 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:31.273759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cbd049-8f0a-4905-aa80-26212566d40d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:31.280773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:31.282347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:31.282948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:31.283754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] PATH_INFO: `/v3/projects/c8597a47523048e78c0791720c6e677a/users/3466f9e75a644b04869bf7f460b8642e/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:31.302987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:31.317279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8597a47523048e78c0791720c6e677a, user_id=3466f9e75a644b04869bf7f460b8642e, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:31.325373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:31.350262 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1351/2716] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:31 2022] PUT /identity/v3/projects/c8597a47523048e78c0791720c6e677a/users/3466f9e75a644b04869bf7f460b8642e/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:31.371371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:31.371871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:31.372377 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:31.426007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] MFA Rules not processed for user `3466f9e75a644b04869bf7f460b8642e`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:31.428008 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:31.518429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1366/2717] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:31.533086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:31.535281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:31.659235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b339f261-3491-4dc2-bf48-d1d097e9e216 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:31.668133 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec59d7f-bae0-46f6-b825-ab386cea2626 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:31.669148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec59d7f-bae0-46f6-b825-ab386cea2626 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:31.669611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec59d7f-bae0-46f6-b825-ab386cea2626 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:31.670076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec59d7f-bae0-46f6-b825-ab386cea2626 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:31.684612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec59d7f-bae0-46f6-b825-ab386cea2626 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:31.687358 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec59d7f-bae0-46f6-b825-ab386cea2626 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:31.705277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1352/2718] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:32.078734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:32.226890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80525c4-12e0-4ffa-9404-1307be5ab755 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:32.236480 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:32.237629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:32.238110 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:32.238429 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] PATH_INFO: `/v3/users/3466f9e75a644b04869bf7f460b8642e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:32.240155 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] RBAC: Authorizing `identity:delete_user(user_id=3466f9e75a644b04869bf7f460b8642e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:32.242290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:32.320419 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3466f9e75a644b04869bf7f460b8642e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:32.321356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3466f9e75a644b04869bf7f460b8642e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:32.322091 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3466f9e75a644b04869bf7f460b8642e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:32.333435 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3466f9e75a644b04869bf7f460b8642e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:32.342603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3466f9e75a644b04869bf7f460b8642e'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:32.349646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:32.365896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1367/2719] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:32 2022] DELETE /identity/v3/users/3466f9e75a644b04869bf7f460b8642e => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:32.371013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:32.371352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:32.371736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:32.393159 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:32.394421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:32.464998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1353/2720] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:32.480350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:32.482077 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:32.556661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21d709e-ed92-4f2a-a079-31ff42102cee admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:32.561134 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cea40b-1dd8-4576-bf73-e34d215bb99c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:32.561851 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cea40b-1dd8-4576-bf73-e34d215bb99c service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:32.562718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cea40b-1dd8-4576-bf73-e34d215bb99c service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:32.563004 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cea40b-1dd8-4576-bf73-e34d215bb99c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:32.574456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cea40b-1dd8-4576-bf73-e34d215bb99c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:32.576300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cea40b-1dd8-4576-bf73-e34d215bb99c service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:32.587979 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1368/2721] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:33.023102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.025125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:33.047434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1dfdb3-53d1-46b3-84d4-9ae60f79a3f5 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.069292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.070746 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.071459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.071941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] PATH_INFO: `/v3/projects/c8597a47523048e78c0791720c6e677a` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.074880 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8597a47523048e78c0791720c6e677a)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.085994 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.155537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8597a47523048e78c0791720c6e677a', 'request_id': 'req-07926e37-a2df-4da2-8578-c624c1c82f60'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:33.163145 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8597a47523048e78c0791720c6e677a', 'request_id': 'req-07926e37-a2df-4da2-8578-c624c1c82f60'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:33.174082 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8597a47523048e78c0791720c6e677a', 'request_id': 'req-07926e37-a2df-4da2-8578-c624c1c82f60'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:33.177079 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:33.204957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] The token cache is being invalidate because project c8597a47523048e78c0791720c6e677a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:33.205281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:33.206007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-07926e37-a2df-4da2-8578-c624c1c82f60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:33.208041 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1354/2722] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:33 2022] DELETE /identity/v3/projects/c8597a47523048e78c0791720c6e677a => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:33.213559 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.213895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.214221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.253300 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:33.255236 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:33.383166 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1369/2723] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:33.397224 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.397710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.398192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.433216 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:33.435586 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:33.451334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1355/2724] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:33.458131 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.476322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ab7baf-5420-48f1-8253-4500960b942e None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.485401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.486376 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.486848 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.487269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.496739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:33.496739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.496739 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.517018 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1370/2725] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:47:33.522767 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.541267 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afdddab-20b4-4d91-ae81-e9151c976b3a None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.551264 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.552327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.552832 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.553362 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.554563 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.558353 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.605306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1356/2726] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:33 2022] POST /identity/v3/projects => generated 371 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:33.610939 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.623943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567a2aff-e270-42c9-9fdd-dc1d4ae491b3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.693611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.694238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.694517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.694789 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.695437 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.697518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.723069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1371/2727] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:33 2022] POST /identity/v3/users => generated 286 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:47:33.727800 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.740257 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bd2084-8cec-4e42-b3a4-614e257d28f7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.745601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.746238 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.746525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.746796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.747480 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.749255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.761975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1357/2728] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:33.767238 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.778605 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd2b12-bb92-4b82-80c9-fa55dd4360b3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.784114 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f7f0b5-fc17-4a97-a07c-8d6808f9cafa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.784783 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f7f0b5-fc17-4a97-a07c-8d6808f9cafa admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.785274 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f7f0b5-fc17-4a97-a07c-8d6808f9cafa admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.785551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f7f0b5-fc17-4a97-a07c-8d6808f9cafa admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.786227 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f7f0b5-fc17-4a97-a07c-8d6808f9cafa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.787885 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f7f0b5-fc17-4a97-a07c-8d6808f9cafa admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.789594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:33.804044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1372/2729] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:33 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:47:33.808940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:33.820460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7477771-c259-40a6-a4de-78c6c30a517c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:33.825448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:33.826137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.826432 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.826709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] PATH_INFO: `/v3/projects/8cc7eb31400941b9b394f381f947781e/users/fc3e37f902874b01affd4b66be9187bb/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.845297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cc7eb31400941b9b394f381f947781e, user_id=fc3e37f902874b01affd4b66be9187bb, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:33.848590 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:33.867308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1358/2730] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:33 2022] PUT /identity/v3/projects/8cc7eb31400941b9b394f381f947781e/users/fc3e37f902874b01affd4b66be9187bb/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:33.879162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:33.879508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:33.879750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:33.913990 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] MFA Rules not processed for user `fc3e37f902874b01affd4b66be9187bb`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:33.915313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:34.004055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1373/2731] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:34.009323 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.096554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eec953-047d-4011-8651-bcc12ec31ee1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.103121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.103915 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.104375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.104783 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.105830 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.108197 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.149520 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1359/2732] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:34 2022] POST /identity/v3/projects => generated 371 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:34.154456 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.167637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bd85ab-6286-4636-8983-7fefb8bf6f17 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.232703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.233333 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.233615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.233954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.234582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.236659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.261249 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1374/2733] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:34 2022] POST /identity/v3/users => generated 292 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:34.267475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.279472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47444d44-a74f-4e95-b7be-cf02dc573233 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.286367 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.287134 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.287468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.287780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.288538 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.290388 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.306206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1360/2734] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:34 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:34.312523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.326090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224be82c-983f-4fad-a69c-6d477db2afbe admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.335710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.337089 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.337926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.338353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] PATH_INFO: `/v3/projects/671a7ea632e5444492a628b357879faf/users/28faa2c5513f4195b0d674df4087b9e3/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.358330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=671a7ea632e5444492a628b357879faf, user_id=28faa2c5513f4195b0d674df4087b9e3, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.362081 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.378452 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1375/2735] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:34 2022] PUT /identity/v3/projects/671a7ea632e5444492a628b357879faf/users/28faa2c5513f4195b0d674df4087b9e3/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:34.382963 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.394270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3f447-ae3d-4e06-944e-fc95c6686aac admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.467346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad54d14-632d-423f-9f3d-33a4e788c5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.467971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad54d14-632d-423f-9f3d-33a4e788c5a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.468843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad54d14-632d-423f-9f3d-33a4e788c5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.469108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad54d14-632d-423f-9f3d-33a4e788c5a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.469757 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad54d14-632d-423f-9f3d-33a4e788c5a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.471414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad54d14-632d-423f-9f3d-33a4e788c5a3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.483595 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1361/2736] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:34 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:34.489388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.501415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a1e848-6afe-4d59-8231-6ef0df74f9bc admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.507364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.508886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.509582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.509943 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] PATH_INFO: `/v3/projects/671a7ea632e5444492a628b357879faf/users/28faa2c5513f4195b0d674df4087b9e3/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.512520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] RBAC: Authorizing `identity:create_grant(project_id=671a7ea632e5444492a628b357879faf, user_id=28faa2c5513f4195b0d674df4087b9e3, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.516184 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.529832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1376/2737] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:34 2022] PUT /identity/v3/projects/671a7ea632e5444492a628b357879faf/users/28faa2c5513f4195b0d674df4087b9e3/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:34.542697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.542975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.543252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.551331 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:34.585112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] MFA Rules not processed for user `28faa2c5513f4195b0d674df4087b9e3`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:34.586400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:34.702366 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1362/2738] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:34 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:34.714567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.716386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:34.767293 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:34.795597 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1a0ae3-2d9e-4292-8883-d3279f9a194e admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.802550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.803669 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.804197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.804778 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:34.820718 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:34.822371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:34.833773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1377/2739] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:34 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:34.922163 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:34.934968 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638b3c6-c0f7-414f-aa1e-033bd3baa91e None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:34.942744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1181a8-4334-4736-8886-39a4fe067493 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:34.943412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1181a8-4334-4736-8886-39a4fe067493 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:34.943688 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1181a8-4334-4736-8886-39a4fe067493 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:34.943959 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1181a8-4334-4736-8886-39a4fe067493 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:35.021112 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1181a8-4334-4736-8886-39a4fe067493 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:35.022488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1181a8-4334-4736-8886-39a4fe067493 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:35.033265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1363/2740] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:34 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:44.829021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=9 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:45.028223 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=61 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:51.530236 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:51.533548 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:51.615918 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290645c8-18b0-4204-90ad-0f7993588252 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:51.621277 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:51.621941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:51.622228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:51.622503 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] PATH_INFO: `/v3/users/dd8fc8ea008f4b8a9aef38f0f3d075bc` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:51.623706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd8fc8ea008f4b8a9aef38f0f3d075bc)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:51.625773 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:51.686229 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd8fc8ea008f4b8a9aef38f0f3d075bc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:51.687213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd8fc8ea008f4b8a9aef38f0f3d075bc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:51.688398 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd8fc8ea008f4b8a9aef38f0f3d075bc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:51.702354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd8fc8ea008f4b8a9aef38f0f3d075bc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:51.719358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd8fc8ea008f4b8a9aef38f0f3d075bc'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:51.746016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1378/2741] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:47:51 2022] DELETE /identity/v3/users/dd8fc8ea008f4b8a9aef38f0f3d075bc => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:51.751255 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:51.751877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:51.751877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:51.774804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:51.776229 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:51.866169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1364/2742] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:51.879974 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:51.881479 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:51.957626 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002924cb-ba2a-42f3-8b45-ab3bdaa88cad admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:51.962569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20aa8609-75d5-4603-b2ce-5794b6d7543c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:51.963251 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20aa8609-75d5-4603-b2ce-5794b6d7543c service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:51.963708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20aa8609-75d5-4603-b2ce-5794b6d7543c service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:51.964149 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20aa8609-75d5-4603-b2ce-5794b6d7543c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:51.976144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20aa8609-75d5-4603-b2ce-5794b6d7543c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:51.977500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20aa8609-75d5-4603-b2ce-5794b6d7543c service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:51.989558 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1379/2743] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:47:52.337072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:52.340006 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:52.366906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e831bd1-3104-49c7-8481-b6cc799f02c0 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:52.379949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:52.381508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:52.382312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:52.383039 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] PATH_INFO: `/v3/projects/8be5e93b1f2a4ba5b016fe0136a3b8a7` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:52.385535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8be5e93b1f2a4ba5b016fe0136a3b8a7)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:52.389712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:52.466544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8be5e93b1f2a4ba5b016fe0136a3b8a7', 'request_id': 'req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:52.477780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8be5e93b1f2a4ba5b016fe0136a3b8a7', 'request_id': 'req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:52.491440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8be5e93b1f2a4ba5b016fe0136a3b8a7', 'request_id': 'req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:52.510557 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] The token cache is being invalidate because project 8be5e93b1f2a4ba5b016fe0136a3b8a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:47:52.510971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:52.511830 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-abbdc80a-427a-4f51-b9c6-1faa9ebeb812 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:47:52.513816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1365/2744] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:47:52 2022] DELETE /identity/v3/projects/8be5e93b1f2a4ba5b016fe0136a3b8a7 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:47:52.522353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:52.522763 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:52.523270 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:52.559929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:52.563719 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:52.597594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:52.682447 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1380/2745] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:52.692414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:52.692763 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:52.693088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:52.715976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:52.717332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:52.726492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1366/2746] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:52.731029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:52.741973 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093f6dc7-f2f7-4362-adad-5d4bed144a84 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:52.746994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:52.747630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:52.747914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:52.748188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:52.748899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:47:52.749211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:52.751144 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:52.764199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1381/2747] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:47:52.771371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:52.789973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b3055b-dca8-4e04-8648-401a4b743879 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:52.798918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:52.799864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:52.800369 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:52.800853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:52.801921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:52.805601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:52.849437 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1367/2748] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:52 2022] POST /identity/v3/projects => generated 383 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:52.852335 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:52.869495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d074f-3374-4f3f-bae7-4ada82789079 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:52.977683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:52.978442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:52.978760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:52.978978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:52.979678 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:52.981958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:53.026910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1382/2749] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:52 2022] POST /identity/v3/users => generated 292 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:53.031898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:53.043878 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b476d-832e-4d68-a916-647bd57e0020 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:53.051307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:53.052374 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:53.052785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:53.053305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:53.054459 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:53.057596 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:53.077872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1368/2750] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:47:53.082532 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:53.099258 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd5255-e725-4e43-8885-d0c838bcd4f1 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:53.105574 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecf16f1-d896-4bef-948d-9bc936adfc33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:53.106242 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf16f1-d896-4bef-948d-9bc936adfc33 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:53.106695 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf16f1-d896-4bef-948d-9bc936adfc33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:53.107058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf16f1-d896-4bef-948d-9bc936adfc33 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:53.107924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecf16f1-d896-4bef-948d-9bc936adfc33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:53.110661 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecf16f1-d896-4bef-948d-9bc936adfc33 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:53.129604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1383/2751] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:47:53.135673 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:53.151274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612c3b6-05da-4217-be1e-321522381400 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:53.160690 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:53.161948 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:53.162727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:53.163284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] PATH_INFO: `/v3/projects/673afe3c625d419e89fe80f3db4574fb/users/a11fb6edaae94f8ba2214e5ec0c65972/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:53.194823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] RBAC: Authorizing `identity:create_grant(project_id=673afe3c625d419e89fe80f3db4574fb, user_id=a11fb6edaae94f8ba2214e5ec0c65972, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:53.199700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:53.225988 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1369/2752] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:47:53 2022] PUT /identity/v3/projects/673afe3c625d419e89fe80f3db4574fb/users/a11fb6edaae94f8ba2214e5ec0c65972/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:47:53.232365 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fb4104-4bfb-4886-8bda-8be364364307 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:53.232852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fb4104-4bfb-4886-8bda-8be364364307 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:53.233356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fb4104-4bfb-4886-8bda-8be364364307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:53.269840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-d7fb4104-4bfb-4886-8bda-8be364364307 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:53.269840 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d7fb4104-4bfb-4886-8bda-8be364364307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:53.378921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1384/2753] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:47:53.389779 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:53.392474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:53.417711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:53.464951 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30933b0-00c5-4ec6-8319-2df404abd485 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:53.469284 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:53.469926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:53.470207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:53.470481 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:53.482482 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:53.484527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:53.495699 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1370/2754] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:47:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:47:59.729630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ac4105-b016-4826-ae61-517963160f78 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:59.730516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ac4105-b016-4826-ae61-517963160f78 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:59.730667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ac4105-b016-4826-ae61-517963160f78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:59.773932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-84ac4105-b016-4826-ae61-517963160f78 None None] MFA Rules not processed for user `a11fb6edaae94f8ba2214e5ec0c65972`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:47:59.775606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-84ac4105-b016-4826-ae61-517963160f78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:47:59.808521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:47:59.882391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1385/2755] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:47:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:47:59.887622 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:59.900843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617832f-cb96-4e7a-acd2-e7aa10da28ec service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:47:59.909518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:47:59.910919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:47:59.911483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:47:59.912045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:47:59.913292 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:47:59.917076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:47:59.957592 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1371/2756] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:47:59 2022] POST /identity/v3/projects => generated 385 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:47:59.962400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ac4105-b016-4826-ae61-517963160f78 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:47:59.975573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ac4105-b016-4826-ae61-517963160f78 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:00.066099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:00.066734 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:00.067016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:00.067289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:00.067941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:00.070354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:00.099411 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1386/2757] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:47:59 2022] POST /identity/v3/users => generated 299 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:00.104774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:00.119207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8772ae21-0c6f-450f-8076-b96f9d152245 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:00.124973 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:00.126025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:00.126421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:00.126606 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:00.127299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:00.130502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:00.151545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1372/2758] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:00.157105 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:00.171941 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac31fd7-0412-4c7d-b5f7-175b353f3ce6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:00.178421 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:00.180635 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:00.181044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:00.181412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] PATH_INFO: `/v3/projects/de90994b10764efbae4183e20b04b05c/users/4d12448b49194995ad418c4437c1db49/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:00.200710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de90994b10764efbae4183e20b04b05c, user_id=4d12448b49194995ad418c4437c1db49, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:00.204082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:00.221009 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1387/2759] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:00 2022] PUT /identity/v3/projects/de90994b10764efbae4183e20b04b05c/users/4d12448b49194995ad418c4437c1db49/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:00.225534 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:00.237548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8b53f-9cc5-426b-b167-8e6e9064d304 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:00.308518 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:00.309302 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:00.309585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:00.309861 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:00.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:00.312875 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:00.324967 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1373/2760] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:00.329863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:00.341308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c754d-72f9-486f-8d90-f1a7422080f5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:00.346369 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab7ae7-a1be-4392-bb85-20e96c700938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:00.347073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab7ae7-a1be-4392-bb85-20e96c700938 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:00.347358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab7ae7-a1be-4392-bb85-20e96c700938 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:00.347641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab7ae7-a1be-4392-bb85-20e96c700938 admin admin] PATH_INFO: `/v3/projects/de90994b10764efbae4183e20b04b05c/users/4d12448b49194995ad418c4437c1db49/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:00.350084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab7ae7-a1be-4392-bb85-20e96c700938 admin admin] RBAC: Authorizing `identity:create_grant(project_id=de90994b10764efbae4183e20b04b05c, user_id=4d12448b49194995ad418c4437c1db49, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:00.352915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab7ae7-a1be-4392-bb85-20e96c700938 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:00.365359 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1388/2761] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:00 2022] PUT /identity/v3/projects/de90994b10764efbae4183e20b04b05c/users/4d12448b49194995ad418c4437c1db49/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:02.497564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:02.501338 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:02.532503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa64953-3730-4666-ab97-1894d74347db admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:02.665901 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:02.666618 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:02.667508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:02.667843 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] PATH_INFO: `/v3/users/41882a2a38f342cb8badc204fc161108` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:02.669899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41882a2a38f342cb8badc204fc161108)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:02.672868 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:02.716398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:02.748906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41882a2a38f342cb8badc204fc161108'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:02.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41882a2a38f342cb8badc204fc161108'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:02.775660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41882a2a38f342cb8badc204fc161108'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:02.777177 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41882a2a38f342cb8badc204fc161108'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:02.778921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41882a2a38f342cb8badc204fc161108'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:02.816811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1374/2762] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:02 2022] DELETE /identity/v3/users/41882a2a38f342cb8badc204fc161108 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:02.822806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:02.823521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:02.823521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:02.852332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:02.854143 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:02.955592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1389/2763] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:02.970063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:02.972945 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:03.054509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb913ef3-4185-4f2a-9e85-995165edd1c7 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:03.059293 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a274e43-8d9e-4170-84e4-661335c25894 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:03.060072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a274e43-8d9e-4170-84e4-661335c25894 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:03.060446 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a274e43-8d9e-4170-84e4-661335c25894 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:03.060734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a274e43-8d9e-4170-84e4-661335c25894 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:03.086791 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a274e43-8d9e-4170-84e4-661335c25894 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:03.089682 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a274e43-8d9e-4170-84e4-661335c25894 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:03.109155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1375/2764] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:03.789717 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:03.792602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:03.831938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566d688-d652-43c9-9500-44eb37593b04 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:03.840357 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:03.841871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:03.842598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:03.843844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] PATH_INFO: `/v3/projects/303ece81a3e84ae6b979c9bce3847c72` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:03.846610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] RBAC: Authorizing `identity:delete_project(project_id=303ece81a3e84ae6b979c9bce3847c72)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:03.850870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:03.864458 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:03.977767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '303ece81a3e84ae6b979c9bce3847c72', 'request_id': 'req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:03.995286 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '303ece81a3e84ae6b979c9bce3847c72', 'request_id': 'req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:04.015744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '303ece81a3e84ae6b979c9bce3847c72', 'request_id': 'req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:04.036028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] The token cache is being invalidate because project 303ece81a3e84ae6b979c9bce3847c72 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:04.037016 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:04.038711 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-7ef8125e-215d-4e3c-b6c0-bf1dfe89c52c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:04.049021 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1390/2765] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:03 2022] DELETE /identity/v3/projects/303ece81a3e84ae6b979c9bce3847c72 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:04.052785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:04.053710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:04.053710 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:04.107491 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:04.110601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:04.268844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:04.269607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:04.270236 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:04.272098 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1376/2766] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:04.324170 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:04.326518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:04.377178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1391/2767] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:04.378175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:04.405141 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5089f67-62a7-4329-9ccb-914cc2ffeebb None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:04.418138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:04.419315 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:04.419753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:04.422154 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:04.423525 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:48:04.424609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:04.427692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:04.495930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1377/2768] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:48:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:48:04.498870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:04.535329 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f804850d-888d-40d8-a65e-f4b13eaafab6 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:04.552707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:04.553911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:04.554810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:04.556352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:04.558228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:04.570056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:04.682598 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:04.686628 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1392/2769] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:04 2022] POST /identity/v3/projects => generated 373 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:04.744931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d90a77-5516-4b4e-8919-944f75460a4c admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:04.972619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:05.072288 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:05.074475 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:05.075300 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:05.075955 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:05.078317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:05.089252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:05.162299 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1378/2770] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:04 2022] POST /identity/v3/users => generated 287 bytes in 480 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:05.171994 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:05.216029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5e1145-83e5-4b0b-a099-d7c80b92baba admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:05.253183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:05.254933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:05.255582 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:05.256230 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:05.258451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:05.262834 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:05.323614 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1393/2771] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:05 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:05.325010 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:05.346700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5f453-cd7f-4d62-9307-c9fe5d707896 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:05.359899 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f9b1dc-a30f-46f0-be2a-80b55d8885dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:05.360573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f9b1dc-a30f-46f0-be2a-80b55d8885dc admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:05.360862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f9b1dc-a30f-46f0-be2a-80b55d8885dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:05.361142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f9b1dc-a30f-46f0-be2a-80b55d8885dc admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:05.361821 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f9b1dc-a30f-46f0-be2a-80b55d8885dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:05.363991 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f9b1dc-a30f-46f0-be2a-80b55d8885dc admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:05.396864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1379/2772] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:05 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:05.398520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:05.446691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb00eb08-c99c-4024-8504-01722f666492 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:05.465533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:05.466221 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:05.466506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:05.466787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] PATH_INFO: `/v3/projects/a3df9bc7e5f54e938a0ab2bdc841ecd3/users/3059ad20d01b49adbb6dbb8ca75e451c/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:05.521223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3df9bc7e5f54e938a0ab2bdc841ecd3, user_id=3059ad20d01b49adbb6dbb8ca75e451c, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:05.533084 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:05.571648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1394/2773] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:05 2022] PUT /identity/v3/projects/a3df9bc7e5f54e938a0ab2bdc841ecd3/users/3059ad20d01b49adbb6dbb8ca75e451c/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:05.578251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee46219-4618-4a2a-8458-7ace105554a1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:05.578717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee46219-4618-4a2a-8458-7ace105554a1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:05.579166 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee46219-4618-4a2a-8458-7ace105554a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:05.670245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8ee46219-4618-4a2a-8458-7ace105554a1 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:05.672709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8ee46219-4618-4a2a-8458-7ace105554a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:05.876301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1380/2774] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:05.886243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:05.888520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:06.024358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6de205-2fc4-43ae-9bbc-5652f0f1b1f7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:06.028741 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:06.029552 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:06.029944 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:06.030358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:06.051823 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:06.055392 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:06.090035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1395/2775] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:08.736371 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f511c5-06d6-453b-a365-d2c53a6d96fc None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:08.737799 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f511c5-06d6-453b-a365-d2c53a6d96fc None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:08.738555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f511c5-06d6-453b-a365-d2c53a6d96fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:08.793398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-34f511c5-06d6-453b-a365-d2c53a6d96fc None None] MFA Rules not processed for user `4d12448b49194995ad418c4437c1db49`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:08.795642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-34f511c5-06d6-453b-a365-d2c53a6d96fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:09.069848 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1381/2776] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:08 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:09.099633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:09.102938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:09.253193 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:09.312932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c48722-3de6-4586-8ca5-cdb22a5e9420 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:09.319243 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7bd8b2ff3ce14290ac0cf476021e3496', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:09.320576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:09.321252 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:09.321999 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:09.326884 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:09.488950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:09.493684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:09.518325 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1396/2777] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:09.725226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9c67ac-4fcb-40ab-80ec-7fedd5ec0a63 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:09.725749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9c67ac-4fcb-40ab-80ec-7fedd5ec0a63 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:09.726204 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9c67ac-4fcb-40ab-80ec-7fedd5ec0a63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:09.851413 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:09.860820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-ba9c67ac-4fcb-40ab-80ec-7fedd5ec0a63 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:09.863168 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ba9c67ac-4fcb-40ab-80ec-7fedd5ec0a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:10.022500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1382/2778] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:48:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:10.091401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:10.094047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:10.317490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8e8134-cb03-4343-a431-8e3b29cb55ef service glance] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:10.329540 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:10.331058 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:10.331640 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:10.335427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:10.365788 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:10.372135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:10.389354 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1397/2779] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:48:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:48:14.520565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:14.521129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:14.521503 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:14.572827 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] MFA Rules not processed for user `3059ad20d01b49adbb6dbb8ca75e451c`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:14.574533 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:14.780230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1383/2780] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:14.782126 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:14.816203 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ec1ab-aabc-43f9-be0d-5946c1b30032 service swift] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:14.828510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:14.830241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:14.830712 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:14.831702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:14.832852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:14.839887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:14.961978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1398/2781] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:14 2022] POST /identity/v3/projects => generated 371 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:14.963312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:14.996352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee5097c-7578-4552-9cc9-20f2753cf358 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:15.284798 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:15.285846 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:15.286239 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:15.286613 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:15.288630 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:15.301476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:15.392510 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:15.404346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1384/2782] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:14 2022] POST /identity/v3/users => generated 292 bytes in 442 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:15.411483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5447e-fd8b-4973-b95f-526a46819d09 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:15.417181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:15.418102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:15.418615 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:15.419121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:15.422075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:15.424197 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:15.455282 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:15.467756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1399/2783] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:15.471386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b3369-3b83-4043-b9bf-9e89a0a8d0c2 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:15.479744 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:15.481115 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:15.481509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:15.482042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] PATH_INFO: `/v3/projects/643d5812093148458f0b699a07a15805/users/08d7b062eb5a4c978c1c28d79a39ba1b/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:15.534025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] RBAC: Authorizing `identity:create_grant(project_id=643d5812093148458f0b699a07a15805, user_id=08d7b062eb5a4c978c1c28d79a39ba1b, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:15.548736 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:15.573523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:15.594285 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1385/2784] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:15 2022] PUT /identity/v3/projects/643d5812093148458f0b699a07a15805/users/08d7b062eb5a4c978c1c28d79a39ba1b/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:15.598204 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:15.616553 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50096689-e7de-4ede-9c44-47717b3599b0 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:15.674231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:15.746213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2934a17-22ed-4b54-9324-d4a6d69d2d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:15.747157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2934a17-22ed-4b54-9324-d4a6d69d2d01 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:15.747284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2934a17-22ed-4b54-9324-d4a6d69d2d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:15.747681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2934a17-22ed-4b54-9324-d4a6d69d2d01 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:15.748592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2934a17-22ed-4b54-9324-d4a6d69d2d01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:15.750683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2934a17-22ed-4b54-9324-d4a6d69d2d01 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:15.767291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1400/2785] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:15.768834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:15.789845 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09a2890-7597-4040-b709-92d0cfe9d968 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:15.799078 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:15.800032 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:15.800416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:15.800912 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] PATH_INFO: `/v3/projects/643d5812093148458f0b699a07a15805/users/08d7b062eb5a4c978c1c28d79a39ba1b/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:15.808621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] RBAC: Authorizing `identity:create_grant(project_id=643d5812093148458f0b699a07a15805, user_id=08d7b062eb5a4c978c1c28d79a39ba1b, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:15.812440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:15.832169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1386/2786] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:15 2022] PUT /identity/v3/projects/643d5812093148458f0b699a07a15805/users/08d7b062eb5a4c978c1c28d79a39ba1b/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:24.721910 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64afe2fd-b2a9-451b-bdb1-12f894a9ea72 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:24.723653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64afe2fd-b2a9-451b-bdb1-12f894a9ea72 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:24.724405 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64afe2fd-b2a9-451b-bdb1-12f894a9ea72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:24.791043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-64afe2fd-b2a9-451b-bdb1-12f894a9ea72 None None] MFA Rules not processed for user `08d7b062eb5a4c978c1c28d79a39ba1b`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:24.792404 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-64afe2fd-b2a9-451b-bdb1-12f894a9ea72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:24.947010 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1401/2787] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:24 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:24.977138 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:24.980182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:25.193488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0196fb71-13ce-4b5f-ae4e-e84e39272424 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:25.206409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:25.207214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:25.207605 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:25.208114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:25.319132 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:25.320847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:25.335275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1387/2788] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:28.311064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f222d4-37fb-4d61-a49d-45808fac653d None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:28.312694 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f222d4-37fb-4d61-a49d-45808fac653d None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:28.312988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f222d4-37fb-4d61-a49d-45808fac653d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:28.355518 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a7f222d4-37fb-4d61-a49d-45808fac653d None None] MFA Rules not processed for user `3059ad20d01b49adbb6dbb8ca75e451c`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:28.356829 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a7f222d4-37fb-4d61-a49d-45808fac653d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:28.365642 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1402/2789] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:29.007890 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:29.009863 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:29.202768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0b8760-5a8f-4aac-a663-9ab2d56c3431 service nova] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:29.210206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:29.211621 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:29.212281 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:29.212795 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] PATH_INFO: `/v3/users/fc3e37f902874b01affd4b66be9187bb` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:29.214402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc3e37f902874b01affd4b66be9187bb)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:29.219117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:29.289887 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:29.362340 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc3e37f902874b01affd4b66be9187bb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:29.377528 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc3e37f902874b01affd4b66be9187bb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:29.390885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc3e37f902874b01affd4b66be9187bb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:29.391884 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc3e37f902874b01affd4b66be9187bb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:29.392820 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc3e37f902874b01affd4b66be9187bb'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:29.462333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1388/2790] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:29 2022] DELETE /identity/v3/users/fc3e37f902874b01affd4b66be9187bb => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:29.470374 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:29.471182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:29.471182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:29.569720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:29.572171 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:29.739630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1403/2791] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:29.754326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:29.755923 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:29.914708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d5b876-af1d-4487-80cf-e7ed6e75f0e1 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:29.918922 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:29.919552 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:29.919829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:29.920100 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:29.932556 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:29.946893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:29.966327 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1389/2792] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:30.533007 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:30.537522 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:30.564514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa1a1ec-2ba7-408c-b496-d569db352d5a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:30.575321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:30.576599 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:30.578353 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:30.579042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] PATH_INFO: `/v3/projects/8cc7eb31400941b9b394f381f947781e` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:30.581681 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cc7eb31400941b9b394f381f947781e)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:30.586346 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:30.669560 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:30.697192 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cc7eb31400941b9b394f381f947781e', 'request_id': 'req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:30.709247 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cc7eb31400941b9b394f381f947781e', 'request_id': 'req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:30.729093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cc7eb31400941b9b394f381f947781e', 'request_id': 'req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:30.758340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] The token cache is being invalidate because project 8cc7eb31400941b9b394f381f947781e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:30.758849 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:30.760035 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-299b2e8b-8c05-4344-a7dc-b65fa9fe8aa1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:30.763266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1404/2793] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:30 2022] DELETE /identity/v3/projects/8cc7eb31400941b9b394f381f947781e => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:30.769921 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:30.788212 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373c1dc8-8983-4e4c-9a83-5192420404ce service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:30.906933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:30.909007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:30.909397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:30.909839 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] PATH_INFO: `/v3/users/28faa2c5513f4195b0d674df4087b9e3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:30.911695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28faa2c5513f4195b0d674df4087b9e3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:30.914076 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:30.986536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:30.989265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28faa2c5513f4195b0d674df4087b9e3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.008101 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28faa2c5513f4195b0d674df4087b9e3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.024387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28faa2c5513f4195b0d674df4087b9e3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.026398 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28faa2c5513f4195b0d674df4087b9e3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.027226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28faa2c5513f4195b0d674df4087b9e3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.055458 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1390/2794] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:30 2022] DELETE /identity/v3/users/28faa2c5513f4195b0d674df4087b9e3 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:31.168580 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:31.168984 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:31.169167 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:31.193034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] MFA Rules not processed for user `a11fb6edaae94f8ba2214e5ec0c65972`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:31.194565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:31.337106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1405/2795] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:31.632743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:31.634751 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:31.767339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac02093-9c78-4b26-9a77-8e92a72f13f9 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:31.771524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:31.773747 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:31.774514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:31.774864 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:31.775460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:31.775938 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] PATH_INFO: `/v3/projects/671a7ea632e5444492a628b357879faf` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:31.778245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=671a7ea632e5444492a628b357879faf)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:31.781982 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:31.861573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '671a7ea632e5444492a628b357879faf', 'request_id': 'req-e4641825-337a-45ba-98ea-c8097078fd00'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.872131 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '671a7ea632e5444492a628b357879faf', 'request_id': 'req-e4641825-337a-45ba-98ea-c8097078fd00'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.887926 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '671a7ea632e5444492a628b357879faf', 'request_id': 'req-e4641825-337a-45ba-98ea-c8097078fd00'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.914430 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c6ed2d-f4c2-438d-94c6-5eceae817472 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:31.917515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] The token cache is being invalidate because project 671a7ea632e5444492a628b357879faf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:31.918064 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.919119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e4641825-337a-45ba-98ea-c8097078fd00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:31.922127 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1391/2796] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:31 2022] DELETE /identity/v3/projects/671a7ea632e5444492a628b357879faf => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:31.927175 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:31.927464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:31.927738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:31.959109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:31.961571 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:32.014185 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.015879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.016438 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.016926 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.019852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:32.038371 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.040486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a39f37-2891-42c3-9e41-8e328c3ff9ea service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.050674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:32.056299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1406/2797] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:48:32.076214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1392/2798] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:32.158771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.159024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.159296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.181706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:32.183139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:32.202264 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1407/2799] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:32.206934 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.219748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f70d5c2-0bdc-4b39-ab52-f51d0f970ff7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.227920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.229023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.229562 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.230033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.231210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:48:32.231755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.235402 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.258330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1393/2800] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:48:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:48:32.263705 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.279442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8448dd5a-2b1a-40a9-844f-4e2f87c1ded7 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.288295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.289214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.289683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.290181 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.291308 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.293648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.336622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:32.357988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1408/2801] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:32 2022] POST /identity/v3/projects => generated 365 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:32.360484 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.374941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eae5361-1f3b-4b7f-a5a8-690441e8b595 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.485633 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.486591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.487164 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.487530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.488447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.490972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.536258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1394/2802] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:32 2022] POST /identity/v3/users => generated 283 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:32.542762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.568698 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6609f807-e817-487b-afc1-30178eb03733 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.577744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.579494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.579886 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.580382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.581536 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.584543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.621585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.625188 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1409/2803] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:32 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:32.653311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.657954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:32.662251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d45a73-b25d-4cfd-ba05-e5e8699600da admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.673048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.674516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.675022 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.675526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.677036 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.696297 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.729453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1395/2804] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:32 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:32.732089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:32.753338 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0113d219-1126-434c-b338-eba7e2d64051 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.773714 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22930938-42f1-4b44-b7d7-de5ec3d472b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.774836 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22930938-42f1-4b44-b7d7-de5ec3d472b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.775416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22930938-42f1-4b44-b7d7-de5ec3d472b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.775893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22930938-42f1-4b44-b7d7-de5ec3d472b4 admin admin] PATH_INFO: `/v3/projects/888589b3431846d9b95c4790ae1d0f5b/users/f8e9f7c7a47c464cadba7a8058c9d601/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.808676 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22930938-42f1-4b44-b7d7-de5ec3d472b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=888589b3431846d9b95c4790ae1d0f5b, user_id=f8e9f7c7a47c464cadba7a8058c9d601, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.812279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f70a3e-2563-4985-9778-73a582c2221b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:32.815147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22930938-42f1-4b44-b7d7-de5ec3d472b4 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.820153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:32.820929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.821288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.821585 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.825373 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:32.846697 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1396/2805] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:32 2022] PUT /identity/v3/projects/888589b3431846d9b95c4790ae1d0f5b/users/f8e9f7c7a47c464cadba7a8058c9d601/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:32.855724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:32.859691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:32.872372 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:32.872717 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:32.872937 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:32.876933 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1410/2806] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:32.922677 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] MFA Rules not processed for user `f8e9f7c7a47c464cadba7a8058c9d601`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:32.928144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:33.075872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1397/2807] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:32 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:33.083457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.104931 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f645b9ac-0d33-4231-a9d9-78078a773836 service placement] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:33.246443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:33.247109 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.247401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.247706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.248426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:33.251555 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:33.295565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1411/2808] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:33 2022] POST /identity/v3/projects => generated 365 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:33.305227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.319148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f38c1d-e506-4912-9df4-abac7911b703 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:33.356155 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:33.406774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:33.407423 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.407709 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.407987 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.408792 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:33.410802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:33.450546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1398/2809] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:33 2022] POST /identity/v3/users => generated 289 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:33.454031 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.473045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33172d77-5fd8-45d7-bb08-b0d5d4f298f6 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:33.482446 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:33.484483 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.484894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.485366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:33.489854 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:33.509545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1412/2810] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:33 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:33.514178 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.529046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15770476-03ba-4589-a6ba-9022edc13883 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:33.540118 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:33.542021 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.542554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.543047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] PATH_INFO: `/v3/projects/219cddbf42a14bc199cdafffbf24606e/users/01f8582ed40c4c5887c5c4b5f65ef866/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.569881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=219cddbf42a14bc199cdafffbf24606e, user_id=01f8582ed40c4c5887c5c4b5f65ef866, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:33.572686 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:33.587303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1399/2811] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:33 2022] PUT /identity/v3/projects/219cddbf42a14bc199cdafffbf24606e/users/01f8582ed40c4c5887c5c4b5f65ef866/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:33.591602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.606697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda5482d-bdac-41c7-bbda-8cff7cf8714a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:33.688040 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:33.703875 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db50a-7078-4108-b240-91fd25f5518d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:33.704945 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db50a-7078-4108-b240-91fd25f5518d admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.705327 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db50a-7078-4108-b240-91fd25f5518d admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.705722 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db50a-7078-4108-b240-91fd25f5518d admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.706751 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db50a-7078-4108-b240-91fd25f5518d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:33.708900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db50a-7078-4108-b240-91fd25f5518d admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:33.728619 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1413/2812] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:33 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:33.736067 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.752580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3734ba-f4dd-46d3-b0f3-7f939f9407d0 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:33.760143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:33.760872 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.761153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.761428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] PATH_INFO: `/v3/projects/219cddbf42a14bc199cdafffbf24606e/users/01f8582ed40c4c5887c5c4b5f65ef866/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.763527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] RBAC: Authorizing `identity:create_grant(project_id=219cddbf42a14bc199cdafffbf24606e, user_id=01f8582ed40c4c5887c5c4b5f65ef866, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:33.767214 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:33.790837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1400/2813] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:33 2022] PUT /identity/v3/projects/219cddbf42a14bc199cdafffbf24606e/users/01f8582ed40c4c5887c5c4b5f65ef866/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:33.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:33.798361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:33.799350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:33.857214 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] MFA Rules not processed for user `01f8582ed40c4c5887c5c4b5f65ef866`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:33.859020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:33.965594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1414/2814] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:33 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:33.978027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:33.980190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:34.054776 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc70ddde-1fdd-4895-b9a8-1c82fd1ad22e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:34.058579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaea2da-47d3-437f-bf6c-af843e007e0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:34.059259 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaea2da-47d3-437f-bf6c-af843e007e0b service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:34.059544 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaea2da-47d3-437f-bf6c-af843e007e0b service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:34.059823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaea2da-47d3-437f-bf6c-af843e007e0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:34.070928 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaea2da-47d3-437f-bf6c-af843e007e0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:34.072222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaea2da-47d3-437f-bf6c-af843e007e0b service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:34.085784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1401/2815] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:33 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:34.781473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:34.870859 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa43985c-9963-4f3a-8434-b3f3a2a90ae3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:34.876948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:34.877622 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:34.877906 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:34.878231 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] PATH_INFO: `/v3/users/f8e9f7c7a47c464cadba7a8058c9d601` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:34.879587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8e9f7c7a47c464cadba7a8058c9d601)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:34.881811 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:34.994967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8e9f7c7a47c464cadba7a8058c9d601'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:34.995611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8e9f7c7a47c464cadba7a8058c9d601'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:35.001148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8e9f7c7a47c464cadba7a8058c9d601'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:35.017743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8e9f7c7a47c464cadba7a8058c9d601'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:35.031063 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-fbda2248-b35d-448a-b94e-f28907081c79 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8e9f7c7a47c464cadba7a8058c9d601'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:35.053627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:35.085595 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1415/2816] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:34 2022] DELETE /identity/v3/users/f8e9f7c7a47c464cadba7a8058c9d601 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:35.091235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:35.093233 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:35.093966 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:35.146720 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:35.147478 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:35.148003 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:35.175891 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:35.177341 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:35.177648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:35.178930 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:35.261520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1416/2817] 10.222.0.26 () {58 vars in 1009 bytes} [Sun Aug 7 01:48:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:35.292689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:35.294817 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:35.309387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1402/2818] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:35.327373 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:35.331352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:35.383137 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:35.441419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a6505f-1f6b-4b6a-b66a-dbc4945d1de2 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:35.442866 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2b9ce6-2ed4-408d-accf-cd3acbcd66cc None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:35.446661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:35.447879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:35.448108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:35.448591 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:35.455533 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:35.456656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:35.457145 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:35.457625 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:35.474818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:35.475986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:35.478137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:35.479566 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:35.496664 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1417/2819] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:48:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 07 01:48:35.497047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1403/2820] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:48:36.353760 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:36.357878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:36.388992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e402ec0-0d7c-4ce6-8a44-09e7eac68971 service swift] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:36.397388 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:36.398892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:36.399495 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:36.400046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:36.404157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:36.418997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:36.421346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:36.429706 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:36.433782 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:36.450858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0aba7f-f20c-48c4-aad3-e5e93b1ad19b service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:36.455672 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1418/2821] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:36.469670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:36.471568 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:36.472227 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:36.473044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] PATH_INFO: `/v3/projects/888589b3431846d9b95c4790ae1d0f5b` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:36.476762 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=888589b3431846d9b95c4790ae1d0f5b)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:36.480420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:36.611048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '888589b3431846d9b95c4790ae1d0f5b', 'request_id': 'req-46b743b6-2ed7-4991-a78f-520f852bda1e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:36.634306 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '888589b3431846d9b95c4790ae1d0f5b', 'request_id': 'req-46b743b6-2ed7-4991-a78f-520f852bda1e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:36.652555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '888589b3431846d9b95c4790ae1d0f5b', 'request_id': 'req-46b743b6-2ed7-4991-a78f-520f852bda1e'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:36.694108 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] The token cache is being invalidate because project 888589b3431846d9b95c4790ae1d0f5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:36.695202 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:36.696378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:36.701444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1404/2822] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:36 2022] DELETE /identity/v3/projects/888589b3431846d9b95c4790ae1d0f5b => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:36.709567 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:36.754683 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d2289-e5ef-4841-9c50-9024eca93591 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:36.917744 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:36.919519 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:36.920124 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:36.920831 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] PATH_INFO: `/v3/users/01f8582ed40c4c5887c5c4b5f65ef866` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:36.923409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] RBAC: Authorizing `identity:delete_user(user_id=01f8582ed40c4c5887c5c4b5f65ef866)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:36.928311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:36.960764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:37.068362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01f8582ed40c4c5887c5c4b5f65ef866'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:37.069713 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01f8582ed40c4c5887c5c4b5f65ef866'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:37.072852 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01f8582ed40c4c5887c5c4b5f65ef866'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:37.091107 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01f8582ed40c4c5887c5c4b5f65ef866'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:37.103966 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8f9ac96d-9fc0-4387-b651-5082f7c11a1c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01f8582ed40c4c5887c5c4b5f65ef866'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:37.135710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1419/2823] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:36 2022] DELETE /identity/v3/users/01f8582ed40c4c5887c5c4b5f65ef866 => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:38.532153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:38.534332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:38.578816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b743b6-2ed7-4991-a78f-520f852bda1e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:38.789659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:38.790691 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:38.790996 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:38.791328 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] PATH_INFO: `/v3/projects/219cddbf42a14bc199cdafffbf24606e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:38.795034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=219cddbf42a14bc199cdafffbf24606e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:38.797335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:38.907585 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:38.909656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '219cddbf42a14bc199cdafffbf24606e', 'request_id': 'req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:38.917275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '219cddbf42a14bc199cdafffbf24606e', 'request_id': 'req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:38.937428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '219cddbf42a14bc199cdafffbf24606e', 'request_id': 'req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:38.964976 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] The token cache is being invalidate because project 219cddbf42a14bc199cdafffbf24606e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:38.965279 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:38.965975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-bb1c9e70-e49e-4d41-b4b3-9ac829ca63b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:38.968550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1405/2824] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:38 2022] DELETE /identity/v3/projects/219cddbf42a14bc199cdafffbf24606e => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:38.979015 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:38.979639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:38.979639 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.012721 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:39.015424 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:39.170026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1420/2825] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:39.173008 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.173343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.173748 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.202044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:39.204038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:39.217610 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1406/2826] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:48:39.224289 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:39.243606 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb2c901-724c-4616-921d-a22a60cccac3 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:39.286644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:39.286644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.286644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.286644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.287610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:48:39.287610 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:39.292768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:39.367045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:39.378407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1421/2827] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:48:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:48:39.394321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619e4429-f6e7-4f9a-a00f-1e2e7288cd29 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:39.409368 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:39.410728 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.411188 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.411640 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.412567 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:39.415097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:39.468046 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1407/2828] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:39 2022] POST /identity/v3/projects => generated 401 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:39.469565 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:39.512577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb45712-753d-4adf-9f85-40a317e96969 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:39.674401 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:39.675523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.676045 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.676524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.677517 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:39.680500 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:39.750520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1422/2829] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:39 2022] POST /identity/v3/users => generated 301 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:39.756675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:39.778835 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bb951b-b238-430b-be7c-615d0ceb8907 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:39.806559 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:39.808019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.808516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.809055 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.814807 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:39.817905 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:39.845810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1408/2830] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:39.848342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:39.866550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac9e50f-7a0e-4099-b88d-4676ddd4f519 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:39.877451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c056a169-69d7-4529-a6de-c260867d01d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:39.878148 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c056a169-69d7-4529-a6de-c260867d01d7 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.878436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c056a169-69d7-4529-a6de-c260867d01d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.878715 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c056a169-69d7-4529-a6de-c260867d01d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.879589 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c056a169-69d7-4529-a6de-c260867d01d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:39.881312 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c056a169-69d7-4529-a6de-c260867d01d7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:39.884123 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:39.903647 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1423/2831] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:39.910826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:39.935734 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef81e25-1509-490d-b307-55446bb18d7f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:39.949275 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:39.950158 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:39.950535 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:39.951347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] PATH_INFO: `/v3/projects/a6145ce0d9d0416f98cff876e35ca150/users/0e4d9f546ff14da19f25e50efc7b18a4/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:39.977573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6145ce0d9d0416f98cff876e35ca150, user_id=0e4d9f546ff14da19f25e50efc7b18a4, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:39.981048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.020656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1409/2832] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:39 2022] PUT /identity/v3/projects/a6145ce0d9d0416f98cff876e35ca150/users/0e4d9f546ff14da19f25e50efc7b18a4/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:40.037801 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.038770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.038770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.078901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] MFA Rules not processed for user `0e4d9f546ff14da19f25e50efc7b18a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:40.081475 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:40.182471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1424/2833] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:40 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:40.187913 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:40.322034 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040eb1e5-b3ed-418b-a2dd-14b2089b5f56 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:40.331989 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:40.332722 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.332997 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.333312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.333925 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:40.335857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.376513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1410/2834] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:40 2022] POST /identity/v3/projects => generated 399 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:40.379621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:40.392276 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40275b51-0a91-4737-aedb-7b6146fa7d2a None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:40.482762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:40.484497 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.485795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.485795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.485795 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:40.487748 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.516278 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1425/2835] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:40 2022] POST /identity/v3/users => generated 306 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:40.522245 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:40.542729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a46d322-1da1-4cb2-91f5-dd3cb0a2dc08 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:40.562804 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:40.564096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.564708 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.565307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.567005 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:40.570695 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.603674 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1411/2836] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:40.610912 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:40.633125 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ef9e31-45a7-4736-a7e2-ed8515c325d7 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:40.642451 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:40.643716 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.644319 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.644671 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] PATH_INFO: `/v3/projects/bfefb464df324250a81f435d7c3849ec/users/d55684c669f04237a4fe400c7817b00f/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.678617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfefb464df324250a81f435d7c3849ec, user_id=d55684c669f04237a4fe400c7817b00f, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:40.683890 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.706014 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1426/2837] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:40 2022] PUT /identity/v3/projects/bfefb464df324250a81f435d7c3849ec/users/d55684c669f04237a4fe400c7817b00f/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:40.710464 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:40.722603 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3171a5-9e9c-4cd8-a4ac-ff1dc3d21b31 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:40.817185 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:40.849834 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348d9f9d-13ee-4c0c-84d6-49aef6e56a96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:40.850932 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348d9f9d-13ee-4c0c-84d6-49aef6e56a96 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.851589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348d9f9d-13ee-4c0c-84d6-49aef6e56a96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.852045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348d9f9d-13ee-4c0c-84d6-49aef6e56a96 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.853240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348d9f9d-13ee-4c0c-84d6-49aef6e56a96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:40.856192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348d9f9d-13ee-4c0c-84d6-49aef6e56a96 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.895312 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1412/2838] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:40 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:40.896476 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:40.912153 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e0c977-3227-4e4a-8c81-3f90c97e06ec admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:40.917954 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:40.918982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.919428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.920182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] PATH_INFO: `/v3/projects/bfefb464df324250a81f435d7c3849ec/users/d55684c669f04237a4fe400c7817b00f/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:40.922752 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfefb464df324250a81f435d7c3849ec, user_id=d55684c669f04237a4fe400c7817b00f, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:40.925843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:40.968285 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1427/2839] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:40 2022] PUT /identity/v3/projects/bfefb464df324250a81f435d7c3849ec/users/d55684c669f04237a4fe400c7817b00f/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:40.981463 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:40.981766 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:40.982028 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:41.046661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] MFA Rules not processed for user `d55684c669f04237a4fe400c7817b00f`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:41.047998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:41.177733 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1413/2840] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:40 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:41.194634 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:41.201141 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:41.291767 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:41.329523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d133c8-81be-4f35-a1f4-543c4c57e508 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:41.333646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:41.334234 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:41.334521 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:41.334794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:41.345916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:41.347412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:41.362986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1428/2841] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:41 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:41.471389 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:41.482601 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba19ca-80e6-41ff-a7bb-a1599d543135 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:41.490496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:41.491149 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:41.491476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:41.491782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:41.584685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:41.584685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:41.600560 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1414/2842] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:41 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:46.459384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:46.461826 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:46.554322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0b9eb-0d3f-4f76-96ce-89c1c1b6e990 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:46.560005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:46.560932 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:46.561508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:46.561690 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:46.563648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:46.574810 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:46.576328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29e6dce-9bf3-4c1a-b71a-9afa42d571cc service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:46.586313 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1429/2843] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:46.649918 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:46.725823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3431936-3586-49ef-bb67-455512f7d17e service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:46.731500 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:46.732153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:46.732477 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:46.732784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] PATH_INFO: `/v3/users/0e4d9f546ff14da19f25e50efc7b18a4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:46.734099 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e4d9f546ff14da19f25e50efc7b18a4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:46.736121 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:46.743716 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:46.809847 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e4d9f546ff14da19f25e50efc7b18a4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:46.822724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e4d9f546ff14da19f25e50efc7b18a4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:46.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e4d9f546ff14da19f25e50efc7b18a4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:46.835240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e4d9f546ff14da19f25e50efc7b18a4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:46.835956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e4d9f546ff14da19f25e50efc7b18a4'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:46.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1415/2844] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:46 2022] DELETE /identity/v3/users/0e4d9f546ff14da19f25e50efc7b18a4 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:46.866054 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:46.866343 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:46.866624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:46.893366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:46.894642 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:46.977781 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1430/2845] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:46.987629 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:46.989237 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:47.059656 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e8ea7-49b5-4de3-a162-122120df70de admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:47.065058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:47.065768 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:47.066056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:47.066334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:47.078531 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:47.080523 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:47.095712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1416/2846] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:47.508771 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:47.510948 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:47.528735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3862152a-fc42-4b51-b22f-ba78616ad467 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:47.548400 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:47.550042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:47.550901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:47.551360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] PATH_INFO: `/v3/projects/a6145ce0d9d0416f98cff876e35ca150` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:47.553898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6145ce0d9d0416f98cff876e35ca150)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:47.554935 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:47.615457 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6145ce0d9d0416f98cff876e35ca150', 'request_id': 'req-68445797-b03c-48c8-9133-e7d13fef3f39'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.623215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6145ce0d9d0416f98cff876e35ca150', 'request_id': 'req-68445797-b03c-48c8-9133-e7d13fef3f39'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.635791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6145ce0d9d0416f98cff876e35ca150', 'request_id': 'req-68445797-b03c-48c8-9133-e7d13fef3f39'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.652402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] The token cache is being invalidate because project a6145ce0d9d0416f98cff876e35ca150 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:47.652844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.653602 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.656047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1431/2847] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:47 2022] DELETE /identity/v3/projects/a6145ce0d9d0416f98cff876e35ca150 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:47.662504 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:47.677171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8ae788-c0a8-4c5d-bb1b-d1a45c521b88 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:47.781125 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:47.781802 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:47.782069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:47.782473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] PATH_INFO: `/v3/users/d55684c669f04237a4fe400c7817b00f` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:47.783957 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d55684c669f04237a4fe400c7817b00f)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:47.786091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:47.829347 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:47.853692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd55684c669f04237a4fe400c7817b00f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.864873 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd55684c669f04237a4fe400c7817b00f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.880329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd55684c669f04237a4fe400c7817b00f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.881508 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd55684c669f04237a4fe400c7817b00f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.882515 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-7df26f5e-ee17-4b45-8a0a-3ca7744bdd88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd55684c669f04237a4fe400c7817b00f'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:47.913956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1417/2848] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:47 2022] DELETE /identity/v3/users/d55684c669f04237a4fe400c7817b00f => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:48.242784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:48.245208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:48.270334 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68445797-b03c-48c8-9133-e7d13fef3f39 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:48.282627 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:48.386208 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:48.387315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:48.387813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:48.388352 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] PATH_INFO: `/v3/projects/bfefb464df324250a81f435d7c3849ec` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:48.391067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfefb464df324250a81f435d7c3849ec)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:48.394942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:48.481650 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfefb464df324250a81f435d7c3849ec', 'request_id': 'req-8fc4116f-910c-4186-b35f-abb493d291af'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:48.494268 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfefb464df324250a81f435d7c3849ec', 'request_id': 'req-8fc4116f-910c-4186-b35f-abb493d291af'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:48.507863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfefb464df324250a81f435d7c3849ec', 'request_id': 'req-8fc4116f-910c-4186-b35f-abb493d291af'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:48.522499 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] The token cache is being invalidate because project bfefb464df324250a81f435d7c3849ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:48.523235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8fc4116f-910c-4186-b35f-abb493d291af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:48.525804 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1432/2849] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:48 2022] DELETE /identity/v3/projects/bfefb464df324250a81f435d7c3849ec => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:48.532879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:48.533270 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:48.533619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:48.557378 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:48.558700 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:48.665584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1418/2850] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:48.673727 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:48.674102 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:48.674436 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:48.699164 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:48.700612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:48.713044 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1433/2851] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:48.718749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:48.730785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2be0e6-1faf-4212-bfb5-fa727bf3e253 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:48.741931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:48.742777 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:48.743111 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:48.743391 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:48.744171 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:48:48.744719 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:48.747317 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:48.763597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1419/2852] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:48:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:48:48.769205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:48.781262 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3873056b-611b-4446-a714-0bff6d4387ee None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:48.789210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:48.790039 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:48.790378 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:48.790724 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:48.791455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:48.793603 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:48.845887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1434/2853] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:48 2022] POST /identity/v3/projects => generated 365 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:48.847727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:48.864350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c57b61-f5e0-4faf-a223-32d009c5a695 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:48.954769 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:48.955815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:48.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:48.956813 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:48.959236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:48.963096 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.008469 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1420/2854] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:48 2022] POST /identity/v3/users => generated 283 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:49.013199 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.051942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916b2b45-24ae-4ab5-8d48-af5612d23784 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.061248 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.062328 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.062780 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.063210 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.065246 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.068651 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.093422 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1435/2855] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:49 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:49.095829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.113506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38be79c-c35e-4696-9912-8942d2f9ee2f admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.122097 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a395f2b5-9047-443d-93b1-7a84e953dff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.123165 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a395f2b5-9047-443d-93b1-7a84e953dff3 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.123715 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a395f2b5-9047-443d-93b1-7a84e953dff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.124090 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a395f2b5-9047-443d-93b1-7a84e953dff3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a395f2b5-9047-443d-93b1-7a84e953dff3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.127026 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a395f2b5-9047-443d-93b1-7a84e953dff3 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.156483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1421/2856] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:49 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:49.158173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.175530 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb8c8fe-5abc-41a0-9525-edf124970695 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.181686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.182600 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.183028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.183427 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] PATH_INFO: `/v3/projects/a9e758cdb4944589bedf4bcdea7c8d50/users/9f8867853e8d40c48a19c0457488aba3/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.202844 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9e758cdb4944589bedf4bcdea7c8d50, user_id=9f8867853e8d40c48a19c0457488aba3, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.206965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.225022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1436/2857] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:49 2022] PUT /identity/v3/projects/a9e758cdb4944589bedf4bcdea7c8d50/users/9f8867853e8d40c48a19c0457488aba3/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:49.239828 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.240474 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.240857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.270419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:49.274176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] MFA Rules not processed for user `9f8867853e8d40c48a19c0457488aba3`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:49.276724 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:49.355148 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1422/2858] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:49.360684 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.405630 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:49.436434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91af4e64-0cbe-40f4-81b9-08e840d80536 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.441787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.442587 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.442880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.443218 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.443929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.446443 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.485283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1437/2859] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:49 2022] POST /identity/v3/projects => generated 365 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:49.490381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.507712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62249b5-36a5-448c-ba31-4d909dae558b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.634321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.635194 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.635565 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.635995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.638143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.640033 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.681944 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1423/2860] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:49 2022] POST /identity/v3/users => generated 289 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:49.683730 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.703395 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c29bd79-5cfa-4ebf-834e-7ab2104b1dda admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.714029 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.715156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.715708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.716195 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.717520 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.720529 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.744762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1438/2861] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:49 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:49.750856 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.769310 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cd4b80-197b-4d15-b357-08df6c59fe0a admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.779521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.780537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.780960 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.781386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] PATH_INFO: `/v3/projects/a0bc71c447e74fbfbdf6af3d81bd350e/users/bb00531a4fd14ed096603b4ae54f57a4/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:49.804505 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0bc71c447e74fbfbdf6af3d81bd350e, user_id=bb00531a4fd14ed096603b4ae54f57a4, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:49.810290 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:49.851829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1424/2862] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:49 2022] PUT /identity/v3/projects/a0bc71c447e74fbfbdf6af3d81bd350e/users/bb00531a4fd14ed096603b4ae54f57a4/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:49.859046 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:49.888120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1d12d-5afe-4190-9fb6-675c850ffd2a admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:49.996838 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc8a8d-98e4-4dc1-ae8e-1dfb31ddabf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:49.997686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc8a8d-98e4-4dc1-ae8e-1dfb31ddabf2 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:49.998115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc8a8d-98e4-4dc1-ae8e-1dfb31ddabf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:49.999115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc8a8d-98e4-4dc1-ae8e-1dfb31ddabf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:50.000002 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc8a8d-98e4-4dc1-ae8e-1dfb31ddabf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:50.001950 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc8a8d-98e4-4dc1-ae8e-1dfb31ddabf2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:50.020112 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1439/2863] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:49 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:50.021837 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:50.038142 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb3d3da-ea9f-4de5-b82e-8da46279d8d2 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:50.046359 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:50.047487 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:50.047942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:50.048472 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] PATH_INFO: `/v3/projects/a0bc71c447e74fbfbdf6af3d81bd350e/users/bb00531a4fd14ed096603b4ae54f57a4/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:50.052037 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0bc71c447e74fbfbdf6af3d81bd350e, user_id=bb00531a4fd14ed096603b4ae54f57a4, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:50.057588 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:50.094530 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1425/2864] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:50 2022] PUT /identity/v3/projects/a0bc71c447e74fbfbdf6af3d81bd350e/users/bb00531a4fd14ed096603b4ae54f57a4/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:50.111489 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:50.111997 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:50.112524 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:50.195178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] MFA Rules not processed for user `bb00531a4fd14ed096603b4ae54f57a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:50.195656 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:50.317172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1440/2865] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:50 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:50.335439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:50.338412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:50.436641 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f80f754-2875-42c7-b368-118297841d86 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:50.443786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:50.444819 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:50.445280 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:50.445740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:50.461349 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:50.464002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:50.470881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:50.495949 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1426/2866] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:50 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:48:50.626043 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:50.646120 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9e9be-a705-4fa6-b2c1-05dcd664bd90 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:50.653060 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:50.653726 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:50.654182 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:50.654466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:50.705224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:50.761005 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:50.765315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:50.775428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:50.778930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:50.789892 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1441/2867] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:50 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:50.966207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539266a-0638-4435-8ff4-3df1d88808df service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:50.994489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:50.996490 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:50.997191 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:50.999554 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:51.004755 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:51.036023 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:51.040521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7dd6fe-5e8d-4631-ab87-6fabf21edb79 service nova] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:51.070593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1427/2868] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:48:56.800022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:56.897909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da2a395-e752-4663-ba22-d4efcbc4a3c1 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:56.906057 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:56.907061 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:56.907516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:56.907964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] PATH_INFO: `/v3/users/9f8867853e8d40c48a19c0457488aba3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:56.909750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f8867853e8d40c48a19c0457488aba3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:56.912026 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:56.986073 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f8867853e8d40c48a19c0457488aba3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:56.987115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f8867853e8d40c48a19c0457488aba3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:56.988027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f8867853e8d40c48a19c0457488aba3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:56.999633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f8867853e8d40c48a19c0457488aba3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.012100 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f8867853e8d40c48a19c0457488aba3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.040916 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1442/2869] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:56 2022] DELETE /identity/v3/users/9f8867853e8d40c48a19c0457488aba3 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:57.046781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:57.047416 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:57.047814 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:57.072849 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:57.074324 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:57.167105 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1428/2870] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:57.178920 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:57.180887 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:57.252295 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:57.259066 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830fa9bb-8a51-4f0c-846d-f91d6f91555b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:57.264632 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:57.265330 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:57.265929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:57.266178 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:57.278284 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:57.280381 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:57.293516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1443/2871] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:48:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:48:57.764160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:57.766440 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:57.788537 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606ffa6-e1a4-40cc-8384-9bdc2c45f055 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:57.801250 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:57.802711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:57.803346 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:57.803889 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] PATH_INFO: `/v3/projects/a9e758cdb4944589bedf4bcdea7c8d50` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:57.806143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9e758cdb4944589bedf4bcdea7c8d50)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:57.809385 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:57.881252 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9e758cdb4944589bedf4bcdea7c8d50', 'request_id': 'req-53537035-e715-4c90-a74f-6f95ed27e53a'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.888643 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9e758cdb4944589bedf4bcdea7c8d50', 'request_id': 'req-53537035-e715-4c90-a74f-6f95ed27e53a'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.899497 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9e758cdb4944589bedf4bcdea7c8d50', 'request_id': 'req-53537035-e715-4c90-a74f-6f95ed27e53a'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.913579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] The token cache is being invalidate because project a9e758cdb4944589bedf4bcdea7c8d50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:57.913818 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.914627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:57.916271 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1429/2872] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:57 2022] DELETE /identity/v3/projects/a9e758cdb4944589bedf4bcdea7c8d50 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:57.920959 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:57.932607 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edd5220-9ded-4624-817c-cdb2c4b3c545 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:58.006190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:58.006914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:58.007269 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:58.007594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] PATH_INFO: `/v3/users/bb00531a4fd14ed096603b4ae54f57a4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:58.009139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb00531a4fd14ed096603b4ae54f57a4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:58.011241 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:58.069955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb00531a4fd14ed096603b4ae54f57a4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.070702 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb00531a4fd14ed096603b4ae54f57a4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.071360 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb00531a4fd14ed096603b4ae54f57a4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.081633 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb00531a4fd14ed096603b4ae54f57a4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.091036 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-dbefcb3c-3adc-4d88-9117-b1d6133e4ab9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb00531a4fd14ed096603b4ae54f57a4'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.112940 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1444/2873] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:48:57 2022] DELETE /identity/v3/users/bb00531a4fd14ed096603b4ae54f57a4 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:58.461457 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:58.463670 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:58.483448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53537035-e715-4c90-a74f-6f95ed27e53a admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:58.488801 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:58.584712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:58.585718 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:58.586176 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:58.586625 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] PATH_INFO: `/v3/projects/a0bc71c447e74fbfbdf6af3d81bd350e` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:58.588384 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0bc71c447e74fbfbdf6af3d81bd350e)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:58.591386 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:58.656419 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0bc71c447e74fbfbdf6af3d81bd350e', 'request_id': 'req-989f5ad4-3a1b-4cff-a03e-391f99d84248'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.663038 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0bc71c447e74fbfbdf6af3d81bd350e', 'request_id': 'req-989f5ad4-3a1b-4cff-a03e-391f99d84248'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.671536 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0bc71c447e74fbfbdf6af3d81bd350e', 'request_id': 'req-989f5ad4-3a1b-4cff-a03e-391f99d84248'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.684494 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] The token cache is being invalidate because project a0bc71c447e74fbfbdf6af3d81bd350e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:48:58.684785 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.685450 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-989f5ad4-3a1b-4cff-a03e-391f99d84248 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:48:58.687002 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1430/2874] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:48:58 2022] DELETE /identity/v3/projects/a0bc71c447e74fbfbdf6af3d81bd350e => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:58.692323 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:58.692655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:58.692978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:58.715156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:58.716288 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:58.802266 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1445/2875] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:58.810726 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:58.811616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:58.811852 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:58.838116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:58.839502 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:58.848576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1431/2876] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:58.853257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:58.864572 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb47f3b6-40c8-4b8c-bef5-59167d1d54bd None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:58.869362 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:58.869976 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:58.870257 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:58.870543 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] PATH_INFO: `/v3/domains` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:58.871215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80178) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:48:58.871523 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:58.873226 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:58.881735 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:58.891291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1446/2877] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:48:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 07 01:48:58.894258 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:58.910666 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492d5bbd-fb91-4f32-a90b-c60b05403c20 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:58.917135 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:58.917774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:58.918058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:58.918335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:58.918998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:58.921660 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:58.960274 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1432/2878] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:58 2022] POST /identity/v3/projects => generated 385 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:48:58.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:58.977855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a99880-f176-447a-a0a4-1f66bc44af11 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.046439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.047086 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.047368 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.047644 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.048336 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.050294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.075119 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1447/2879] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:58 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:59.079583 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.091774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823e96a-308b-40c2-b7d6-2239823ec259 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.097720 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.098341 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.098623 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.098897 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.099555 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.101468 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.113329 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1433/2880] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:59 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:59.117631 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.128367 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5879eea1-f7c9-4d7b-a0af-121bc30046fd admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.133239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffef86e2-a8de-4576-a304-5cb5ef62bb9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.133855 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffef86e2-a8de-4576-a304-5cb5ef62bb9c admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.134137 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffef86e2-a8de-4576-a304-5cb5ef62bb9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.134412 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffef86e2-a8de-4576-a304-5cb5ef62bb9c admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.135071 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffef86e2-a8de-4576-a304-5cb5ef62bb9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.136766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffef86e2-a8de-4576-a304-5cb5ef62bb9c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.150090 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1448/2881] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:59.155524 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.166773 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9fe0d-b3e0-42c2-a061-8c16081c8e58 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.172831 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.173507 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.173786 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.174066 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] PATH_INFO: `/v3/projects/989c05c91583497ba762f9f3018a51ba/users/50c0275ec38c495dac8e24fbcb1f8d0b/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.191397 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=989c05c91583497ba762f9f3018a51ba, user_id=50c0275ec38c495dac8e24fbcb1f8d0b, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.194496 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.214343 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1434/2882] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:59 2022] PUT /identity/v3/projects/989c05c91583497ba762f9f3018a51ba/users/50c0275ec38c495dac8e24fbcb1f8d0b/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:48:59.228418 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.228874 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.229417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.277409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] MFA Rules not processed for user `50c0275ec38c495dac8e24fbcb1f8d0b`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:59.279283 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:59.378473 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1449/2883] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:59.383527 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.431428 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:59.469268 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5002b2-81f5-4be1-ba18-5ec2fe2ccd59 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.474521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.475471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.475840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.476117 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.476857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.478979 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.515047 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1435/2884] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:59 2022] POST /identity/v3/projects => generated 385 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:59.520928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.536042 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12194fe-e55d-49c4-87cd-200da802df76 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.610900 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.611538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.611820 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.612096 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.612790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.614784 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.627806 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:48:59.647770 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1450/2885] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:48:59 2022] POST /identity/v3/users => generated 293 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:48:59.653449 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.668962 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419d1cc-b951-47ea-a4bd-36e71a1310bc admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.675427 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.676455 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.676775 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.677072 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.677784 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.679581 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.693550 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1436/2886] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:59 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:48:59.700209 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.713525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a0d411-dd9a-44bb-8f0b-8e8a4511319c admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.720106 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0f2cc0-826b-4b8a-ac1d-3065fb4bddef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.720913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0f2cc0-826b-4b8a-ac1d-3065fb4bddef admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.721290 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0f2cc0-826b-4b8a-ac1d-3065fb4bddef admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.721659 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0f2cc0-826b-4b8a-ac1d-3065fb4bddef admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.722414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0f2cc0-826b-4b8a-ac1d-3065fb4bddef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.724296 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0f2cc0-826b-4b8a-ac1d-3065fb4bddef admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.737062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1451/2887] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:48:59 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:48:59.741854 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:48:59.753294 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3528e8-ffcb-4197-bc8b-e113f0ed76e5 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:48:59.758375 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:48:59.759051 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] REQUEST_METHOD: `PUT` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.759332 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.759609 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] PATH_INFO: `/v3/projects/49c77524eaa04332af33bd68d3bb4cc0/users/9fdbc8af8d384f2fad7044e9ad97c966/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.777561 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=49c77524eaa04332af33bd68d3bb4cc0, user_id=9fdbc8af8d384f2fad7044e9ad97c966, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:48:59.780584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:48:59.796729 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1437/2888] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:48:59 2022] PUT /identity/v3/projects/49c77524eaa04332af33bd68d3bb4cc0/users/9fdbc8af8d384f2fad7044e9ad97c966/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:48:59.807414 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:48:59.807754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:48:59.808099 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:48:59.854618 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] MFA Rules not processed for user `9fdbc8af8d384f2fad7044e9ad97c966`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:48:59.855986 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:48:59.932768 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1452/2889] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:48:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:48:59.939439 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.026017 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dbe061-dcbc-49c6-8d67-30772c394c9e admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.032102 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.032855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.033488 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.033858 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] PATH_INFO: `/v3/projects` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.034579 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.036659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.071053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1438/2890] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:48:59 2022] POST /identity/v3/projects => generated 383 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:49:00.072220 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.084462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e516bb28-cc42-4865-aac5-3a7f64b8e7a6 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.172121 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.172828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.173157 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.173490 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] PATH_INFO: `/v3/users` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.174085 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.176098 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.204786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1453/2891] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:49:00 2022] POST /identity/v3/users => generated 298 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:49:00.209616 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.222053 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f45924-3190-4612-b4dc-81c415a3d72d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.227817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.228532 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.228817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.229089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.229754 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.231399 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.244521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1439/2892] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:49:00.250179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.261816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bb957-66d8-497a-a999-babee07d704b admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.267761 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.268691 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.269093 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.269417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] PATH_INFO: `/v3/projects/2233b5d61ff347998be58711ae0277a3/users/0825d31e5d694c39aab219162c0114d3/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.294901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2233b5d61ff347998be58711ae0277a3, user_id=0825d31e5d694c39aab219162c0114d3, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.299169 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.322294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1454/2893] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:49:00 2022] PUT /identity/v3/projects/2233b5d61ff347998be58711ae0277a3/users/0825d31e5d694c39aab219162c0114d3/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:00.327521 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.346584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e429bc2e-932f-4abe-9494-3a8083584130 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.427770 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:00.436584 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58f4a57-b22f-4329-8f62-fe0a5c12f337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.437447 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f4a57-b22f-4329-8f62-fe0a5c12f337 admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.438048 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f4a57-b22f-4329-8f62-fe0a5c12f337 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.438365 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58f4a57-b22f-4329-8f62-fe0a5c12f337 admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.439088 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f4a57-b22f-4329-8f62-fe0a5c12f337 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.440844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58f4a57-b22f-4329-8f62-fe0a5c12f337 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.452860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1440/2894] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:49:00 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:49:00.457417 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.471432 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100509f6-801d-408b-85d0-ce33c8829efb admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.480348 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.481358 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.481865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.482292 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] PATH_INFO: `/v3/projects/2233b5d61ff347998be58711ae0277a3/users/0825d31e5d694c39aab219162c0114d3/roles/b0ec0093717c472da3640ec7121208d4` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.489072 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2233b5d61ff347998be58711ae0277a3, user_id=0825d31e5d694c39aab219162c0114d3, role_id=b0ec0093717c472da3640ec7121208d4)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.493051 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.506177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1455/2895] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:49:00 2022] PUT /identity/v3/projects/2233b5d61ff347998be58711ae0277a3/users/0825d31e5d694c39aab219162c0114d3/roles/b0ec0093717c472da3640ec7121208d4 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:49:00.517473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.518303 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.518483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.551456 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] MFA Rules not processed for user `0825d31e5d694c39aab219162c0114d3`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:49:00.552749 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:00.637273 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1441/2896] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:49:00 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:49:00.650322 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:00.652578 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:00.691020 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:00.724539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fefad5b-093a-4987-98b4-fc1d0cd41284 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:00.729133 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:00.729776 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:00.730162 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:00.730442 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:00.799444 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:00.800996 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:00.812439 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1456/2897] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:00 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:02.455073 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:02.473580 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7666b2-3529-4c51-88fd-77eb6a48abf1 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:02.481723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:02.483414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:02.484170 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:02.485235 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:02.599408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:02.602780 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:02.620980 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1442/2898] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:49:03.391796 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:03.403929 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0dc139d-31d2-4c72-a7e7-c87edc38bb45 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:03.408426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:03.409190 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:03.412676 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:03.413138 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:03.431340 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:03.433338 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:03.445428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1457/2899] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:03 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:04.399192 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:04.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:04.471570 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc05d9-44f1-426e-be09-a8bf7daf38a8 service cinder] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:04.478272 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:04.479350 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:04.479811 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:04.480335 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:04.483615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:04.552308 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:04.624013 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:04.627833 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dff9e2-632c-4bd0-82db-0e8cd6589be5 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:04.650130 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1443/2900] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:07.866544 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:08.053228 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77ddc3e-a453-4ea5-a42e-46abae64bc76 service cinder] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:08.065223 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:08.066870 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:08.067794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:08.070494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] PATH_INFO: `/v3/users/50c0275ec38c495dac8e24fbcb1f8d0b` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:08.075822 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=50c0275ec38c495dac8e24fbcb1f8d0b)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:08.080928 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:08.217294 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50c0275ec38c495dac8e24fbcb1f8d0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:08.218696 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50c0275ec38c495dac8e24fbcb1f8d0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:08.219762 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50c0275ec38c495dac8e24fbcb1f8d0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:08.237646 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50c0275ec38c495dac8e24fbcb1f8d0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:08.257667 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50c0275ec38c495dac8e24fbcb1f8d0b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:08.272765 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:08.324409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1458/2901] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:49:07 2022] DELETE /identity/v3/users/50c0275ec38c495dac8e24fbcb1f8d0b => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:08.328091 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:08.330195 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:08.330881 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:08.396931 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:49:08.398412 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:08.567253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1444/2902] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:49:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:49:08.588122 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:08.590621 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:08.604381 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:08.606476 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:08.672067 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e029760-67d2-4c46-b3f9-9d56e2a007fd admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:08.678581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:08.679584 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:08.680034 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:08.680680 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:08.697958 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:08.701921 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:08.719491 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1459/2903] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:08.730356 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212ad473-78a0-4cc9-80c6-ae0abdbe4050 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:08.735153 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:08.735826 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:08.736104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:08.736420 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:08.738448 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:08.844860 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:08.847877 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:08.871619 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1445/2904] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:08 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:49:09.303951 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:09.305802 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:09.330791 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff80b04-868f-4d76-9a95-84cc2723c280 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:09.339001 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:09.339876 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:09.340766 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:09.341139 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] PATH_INFO: `/v3/projects/989c05c91583497ba762f9f3018a51ba` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:09.342509 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] RBAC: Authorizing `identity:delete_project(project_id=989c05c91583497ba762f9f3018a51ba)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:09.345361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:09.439754 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '989c05c91583497ba762f9f3018a51ba', 'request_id': 'req-d6c89f03-153e-4cf1-a252-b68e32a81739'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.449215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '989c05c91583497ba762f9f3018a51ba', 'request_id': 'req-d6c89f03-153e-4cf1-a252-b68e32a81739'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.461978 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '989c05c91583497ba762f9f3018a51ba', 'request_id': 'req-d6c89f03-153e-4cf1-a252-b68e32a81739'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.480551 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] The token cache is being invalidate because project 989c05c91583497ba762f9f3018a51ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:49:09.480992 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.481960 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.484477 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1460/2905] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:49:09 2022] DELETE /identity/v3/projects/989c05c91583497ba762f9f3018a51ba => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:09.490551 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:09.507865 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae24aab0-3b87-46dc-8da1-93f0aef20a49 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:09.516307 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=100 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:09.610855 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:09.611589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:09.611930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:09.612321 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] PATH_INFO: `/v3/users/9fdbc8af8d384f2fad7044e9ad97c966` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:09.613600 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fdbc8af8d384f2fad7044e9ad97c966)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:09.615812 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:09.702992 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fdbc8af8d384f2fad7044e9ad97c966'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.715434 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fdbc8af8d384f2fad7044e9ad97c966'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.730790 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fdbc8af8d384f2fad7044e9ad97c966'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.731940 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fdbc8af8d384f2fad7044e9ad97c966'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.733027 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fdbc8af8d384f2fad7044e9ad97c966'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:09.765698 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1446/2906] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:49:09 2022] DELETE /identity/v3/users/9fdbc8af8d384f2fad7044e9ad97c966 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:10.183311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:10.187022 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:10.207896 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c89f03-153e-4cf1-a252-b68e32a81739 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:10.329816 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:10.330818 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:10.331309 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:10.332174 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] PATH_INFO: `/v3/projects/49c77524eaa04332af33bd68d3bb4cc0` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:10.334423 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=49c77524eaa04332af33bd68d3bb4cc0)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:10.338173 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:10.368364 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:10.423342 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49c77524eaa04332af33bd68d3bb4cc0', 'request_id': 'req-1e3fa958-81b0-495c-ad72-2fd8288e2a54'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.431863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49c77524eaa04332af33bd68d3bb4cc0', 'request_id': 'req-1e3fa958-81b0-495c-ad72-2fd8288e2a54'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.443459 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49c77524eaa04332af33bd68d3bb4cc0', 'request_id': 'req-1e3fa958-81b0-495c-ad72-2fd8288e2a54'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.462279 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] The token cache is being invalidate because project 49c77524eaa04332af33bd68d3bb4cc0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:49:10.464677 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.465841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.473235 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1461/2907] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:49:10 2022] DELETE /identity/v3/projects/49c77524eaa04332af33bd68d3bb4cc0 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:10.476114 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:10.478942 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:10.494790 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:10.497983 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:10.548429 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39a5c5b-e282-4c86-b9c5-cd0c3c108529 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:10.707126 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:10.708492 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:10.709509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:10.710007 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] PATH_INFO: `/v3/users/0825d31e5d694c39aab219162c0114d3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:10.714473 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=0825d31e5d694c39aab219162c0114d3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:10.720184 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:10.736187 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3fa958-81b0-495c-ad72-2fd8288e2a54 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:10.743697 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:10.744287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:10.744953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:10.745321 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:10.747441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:10.765899 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:10.767686 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:10.781880 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1462/2908] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:49:10.836929 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:10.904829 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0825d31e5d694c39aab219162c0114d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.922213 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0825d31e5d694c39aab219162c0114d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.947661 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0825d31e5d694c39aab219162c0114d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.949265 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0825d31e5d694c39aab219162c0114d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:10.950933 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-e097e475-d58e-4e44-98a4-8a96d27da2bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0825d31e5d694c39aab219162c0114d3'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:11.001210 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1447/2909] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:49:10 2022] DELETE /identity/v3/users/0825d31e5d694c39aab219162c0114d3 => generated 0 bytes in 522 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:11.789024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:11.791841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:11.813867 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7a0d2b-c69f-4aed-9c90-e4cd9224b690 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:11.910487 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:11.911366 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:11.911663 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:11.911913 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] PATH_INFO: `/v3/projects/2233b5d61ff347998be58711ae0277a3` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:11.913725 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2233b5d61ff347998be58711ae0277a3)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:11.915794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:11.983382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:12.009538 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2233b5d61ff347998be58711ae0277a3', 'request_id': 'req-c2b35953-e07d-4830-bc47-a249166275b3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:12.020878 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2233b5d61ff347998be58711ae0277a3', 'request_id': 'req-c2b35953-e07d-4830-bc47-a249166275b3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:12.034803 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2233b5d61ff347998be58711ae0277a3', 'request_id': 'req-c2b35953-e07d-4830-bc47-a249166275b3'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:12.055938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] The token cache is being invalidate because project 2233b5d61ff347998be58711ae0277a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:49:12.056554 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:12.057496 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-c2b35953-e07d-4830-bc47-a249166275b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:12.059982 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1463/2910] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:49:11 2022] DELETE /identity/v3/projects/2233b5d61ff347998be58711ae0277a3 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:12.065330 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.066003 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.066129 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.101357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:49:12.102879 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:12.193750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1448/2911] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:49:12.207942 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.208898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.209019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.238860 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:49:12.240128 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:12.250011 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1464/2912] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:49:12.254604 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.267160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feb591c-8b3f-4da1-b266-aa032463f736 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.273569 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.274206 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.274498 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.274782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] PATH_INFO: `/v3/domains` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.281956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80179) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 07 01:49:12.281956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:12.281956 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:12.293460 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1449/2913] 10.222.0.26 () {58 vars in 1153 bytes} [Sun Aug 7 01:49:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 07 01:49:12.298118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.310598 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcd7224-7ce8-47f3-b327-f1aeb0f8d5c4 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.318653 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.319315 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.319594 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.319869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.320573 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:12.322756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:12.364213 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1465/2914] 10.222.0.26 () {60 vars in 1153 bytes} [Sun Aug 7 01:49:12 2022] POST /identity/v3/projects => generated 397 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 07 01:49:12.369251 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.383295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112e81a9-8528-4b9d-b74b-e0ba38f782eb admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.461885 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.462414 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.462701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.463081 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] PATH_INFO: `/v3/users` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.463841 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:12.465930 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:12.498743 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1450/2915] 10.222.0.26 () {60 vars in 1144 bytes} [Sun Aug 7 01:49:12 2022] POST /identity/v3/users => generated 299 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 07 01:49:12.506454 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.524394 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33b159a-6743-4d6c-ad20-5513f382add5 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.530306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.531070 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.531455 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.531815 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] PATH_INFO: `/v3/roles` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.532642 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:12.534471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:12.547579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1466/2916] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:49:12 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 07 01:49:12.553441 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.568998 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57beca9-298f-4c1b-8e81-09a8e2bed472 admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.574782 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16de4e43-9e85-4b23-a5f0-6eb5a63dea8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.575589 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de4e43-9e85-4b23-a5f0-6eb5a63dea8b admin admin] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.575986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de4e43-9e85-4b23-a5f0-6eb5a63dea8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.576999 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16de4e43-9e85-4b23-a5f0-6eb5a63dea8b admin admin] PATH_INFO: `/v3/roles` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.577927 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16de4e43-9e85-4b23-a5f0-6eb5a63dea8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:12.580042 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16de4e43-9e85-4b23-a5f0-6eb5a63dea8b admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:12.593920 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1451/2917] 10.222.0.26 () {58 vars in 1122 bytes} [Sun Aug 7 01:49:12 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 07 01:49:12.601176 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.615798 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f3b5d-88c3-429c-bf08-88f7eab43384 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.621028 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.621871 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] REQUEST_METHOD: `PUT` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.622156 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.622434 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] PATH_INFO: `/v3/projects/918353ffd8ef446fad9a472be8f9e069/users/32958d8e698b432b840ccfa7fa6b443c/roles/e24199f1fbf64edfbef577416cf3c4b2` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.644332 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] RBAC: Authorizing `identity:create_grant(project_id=918353ffd8ef446fad9a472be8f9e069, user_id=32958d8e698b432b840ccfa7fa6b443c, role_id=e24199f1fbf64edfbef577416cf3c4b2)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:12.650064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:12.674384 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1467/2918] 10.222.0.26 () {60 vars in 1483 bytes} [Sun Aug 7 01:49:12 2022] PUT /identity/v3/projects/918353ffd8ef446fad9a472be8f9e069/users/32958d8e698b432b840ccfa7fa6b443c/roles/e24199f1fbf64edfbef577416cf3c4b2 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:12.687711 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.688217 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.688615 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:12.734411 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:12.740174 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] MFA Rules not processed for user `32958d8e698b432b840ccfa7fa6b443c`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:49:12.743181 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:12.854263 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1452/2919] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:49:12 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:49:12.868403 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:12.870224 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:12.978714 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa0223-b145-43c3-83fe-3e670ba7c845 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:12.985297 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec6e98a-d665-40e9-a852-e8262241dff6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f2a3c404c5f840368281af8532b7b46b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:12.986409 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec6e98a-d665-40e9-a852-e8262241dff6 service cinder] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:12.986793 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec6e98a-d665-40e9-a852-e8262241dff6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:12.987075 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec6e98a-d665-40e9-a852-e8262241dff6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:13.002756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec6e98a-d665-40e9-a852-e8262241dff6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:13.005160 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec6e98a-d665-40e9-a852-e8262241dff6 service cinder] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:13.024370 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1468/2920] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:12 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:21.580907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:21.656058 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e972c30-e7d9-47a7-882a-03c339ee6dd7 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:21.660888 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:21.661546 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:21.661830 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:21.662106 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] PATH_INFO: `/v3/users/32958d8e698b432b840ccfa7fa6b443c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:21.663305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] RBAC: Authorizing `identity:delete_user(user_id=32958d8e698b432b840ccfa7fa6b443c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:21.665545 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:21.733453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32958d8e698b432b840ccfa7fa6b443c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:21.744086 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32958d8e698b432b840ccfa7fa6b443c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:21.754242 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32958d8e698b432b840ccfa7fa6b443c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:21.754975 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32958d8e698b432b840ccfa7fa6b443c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:21.755652 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32958d8e698b432b840ccfa7fa6b443c'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:21.778738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1453/2921] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:49:21 2022] DELETE /identity/v3/users/32958d8e698b432b840ccfa7fa6b443c => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:21.785428 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:21.786316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:21.786316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:21.812179 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] MFA Rules not processed for user `a75da33429f640299cbc9644c9a49291`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:49:21.813729 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:21.887356 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:21.932685 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1469/2922] 10.222.0.26 () {58 vars in 958 bytes} [Sun Aug 7 01:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:49:21.943489 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:21.945236 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:21.974753 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:22.074209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ec181f-d730-42c9-81c8-b12534310faa admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:22.080893 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:22.082095 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:22.082483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:22.083056 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:22.101712 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:22.104445 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:22.123611 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1454/2923] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:22.481316 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:22.484901 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:22.506863 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a232d49-4816-4d6f-837a-9a47a3594081 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:22.515655 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:22.516756 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:22.517215 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:22.517710 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] PATH_INFO: `/v3/projects/918353ffd8ef446fad9a472be8f9e069` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:22.519786 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=918353ffd8ef446fad9a472be8f9e069)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:22.523448 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:22.597577 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '918353ffd8ef446fad9a472be8f9e069', 'request_id': 'req-d93eb72d-7a1b-4b89-b891-142958626ed1'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:22.605703 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '918353ffd8ef446fad9a472be8f9e069', 'request_id': 'req-d93eb72d-7a1b-4b89-b891-142958626ed1'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:22.617898 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '918353ffd8ef446fad9a472be8f9e069', 'request_id': 'req-d93eb72d-7a1b-4b89-b891-142958626ed1'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:22.632617 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] The token cache is being invalidate because project 918353ffd8ef446fad9a472be8f9e069 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80178) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:49:22.632894 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:22.633576 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-d93eb72d-7a1b-4b89-b891-142958626ed1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:49:22.635103 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1470/2924] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:49:22 2022] DELETE /identity/v3/projects/918353ffd8ef446fad9a472be8f9e069 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:49:32.114612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=25 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:32.629287 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=39,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:49:34.133256 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:49:34.135198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:34.240573 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d730b-b07f-4471-a7f4-3e3f551eba6a service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:49:34.331305 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:49:34.331935 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:49:34.332219 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:49:34.332526 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:49:34.334471 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:49:34.350326 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:49:34.353693 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:49:34.372209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1455/2925] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:49:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:49:44.364642 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=67 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:50:31.379339 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:50:31.380047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:50:31.380047 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:50:31.406386 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] MFA Rules not processed for user `4814b0fcdd08486bb0f7ae5d8486b9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:50:31.407915 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:50:31.484953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1471/2926] 10.222.0.26 () {58 vars in 967 bytes} [Sun Aug 7 01:50:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:50:31.536810 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:50:31.538738 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:50:31.661593 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9637255-80a4-4868-b2d1-a3a191a16b86 service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:50:31.676136 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7342064620f34dabb93dd10f92f444a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:50:31.676898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:50:31.677253 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:50:31.677540 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:50:31.688816 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:50:31.690627 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:50:31.700898 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1456/2927] 10.222.0.26 () {60 vars in 1384 bytes} [Sun Aug 7 01:50:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 07 01:50:40.296545 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:50:40.300836 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:50:40.406581 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f919a345-7aeb-452e-bde9-a8a9ab02e003 None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:50:40.411299 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:50:40.411967 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:50:40.412281 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:50:40.412566 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] PATH_INFO: `/v3/users/a11fb6edaae94f8ba2214e5ec0c65972` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:50:40.413895 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a11fb6edaae94f8ba2214e5ec0c65972)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:50:40.415949 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:50:40.480118 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a11fb6edaae94f8ba2214e5ec0c65972'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.480909 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a11fb6edaae94f8ba2214e5ec0c65972'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.482416 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a11fb6edaae94f8ba2214e5ec0c65972'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.492759 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a11fb6edaae94f8ba2214e5ec0c65972'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.507361 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a11fb6edaae94f8ba2214e5ec0c65972'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.533787 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1472/2928] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:50:40 2022] DELETE /identity/v3/users/a11fb6edaae94f8ba2214e5ec0c65972 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:50:40.772320 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:50:40.791342 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58cd3f3-315d-49da-b8b1-25f04cec973e service swift] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:50:40.884759 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:50:40.885853 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:50:40.886355 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:50:40.886857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] PATH_INFO: `/v3/projects/673afe3c625d419e89fe80f3db4574fb` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:50:40.890232 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] RBAC: Authorizing `identity:delete_project(project_id=673afe3c625d419e89fe80f3db4574fb)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:50:40.892764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:50:40.947781 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '673afe3c625d419e89fe80f3db4574fb', 'request_id': 'req-84e134f5-35f9-4ca6-87a7-955be2554afc'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.955764 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '673afe3c625d419e89fe80f3db4574fb', 'request_id': 'req-84e134f5-35f9-4ca6-87a7-955be2554afc'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.964986 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '673afe3c625d419e89fe80f3db4574fb', 'request_id': 'req-84e134f5-35f9-4ca6-87a7-955be2554afc'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.978513 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] The token cache is being invalidate because project 673afe3c625d419e89fe80f3db4574fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:50:40.978883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.979576 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:40.981352 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1457/2929] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:50:40 2022] DELETE /identity/v3/projects/673afe3c625d419e89fe80f3db4574fb => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:50:40.986914 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:50:40.988550 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:50:41.003402 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8642822e-cc34-4f7b-a655-c8280a148fb2 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:50:41.027964 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:50:41.073629 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:50:41.074280 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:50:41.074557 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:50:41.074828 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] PATH_INFO: `/v3/users/4d12448b49194995ad418c4437c1db49` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:50:41.076612 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d12448b49194995ad418c4437c1db49)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:50:41.078592 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:50:41.132750 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d12448b49194995ad418c4437c1db49'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.133764 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d12448b49194995ad418c4437c1db49'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.134525 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d12448b49194995ad418c4437c1db49'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.146350 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d12448b49194995ad418c4437c1db49'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.155648 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d12448b49194995ad418c4437c1db49'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.182924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1473/2930] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:50:40 2022] DELETE /identity/v3/users/4d12448b49194995ad418c4437c1db49 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:50:41.398669 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:50:41.401147 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:50:41.424019 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e134f5-35f9-4ca6-87a7-955be2554afc admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:50:41.447862 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:50:41.501325 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:50:41.501866 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:50:41.502179 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:50:41.502483 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] PATH_INFO: `/v3/projects/de90994b10764efbae4183e20b04b05c` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:50:41.503636 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de90994b10764efbae4183e20b04b05c)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:50:41.505675 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:50:41.562577 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de90994b10764efbae4183e20b04b05c', 'request_id': 'req-22fc96f7-9b8b-4000-a8d6-cfcff7470527'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.579409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de90994b10764efbae4183e20b04b05c', 'request_id': 'req-22fc96f7-9b8b-4000-a8d6-cfcff7470527'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.590104 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de90994b10764efbae4183e20b04b05c', 'request_id': 'req-22fc96f7-9b8b-4000-a8d6-cfcff7470527'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.608685 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] The token cache is being invalidate because project de90994b10764efbae4183e20b04b05c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:50:41.609209 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.610277 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-22fc96f7-9b8b-4000-a8d6-cfcff7470527 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:50:41.612907 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1458/2931] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:50:41 2022] DELETE /identity/v3/projects/de90994b10764efbae4183e20b04b05c => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:50:51.178506 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=42,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:50:51.601612 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:51:23.664955 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:51:23.666539 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:23.734953 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fae226-0eff-4839-b94d-86f886f07cc3 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:51:23.792069 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:51:23.792827 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:23.793183 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:23.794108 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:23.796743 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:23.883516 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:51:23.884988 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2dc8b9-9191-4fce-b4df-61e563c490af service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:51:23.898911 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1474/2932] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:51:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:51:24.281466 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0108ea5-13ec-46be-894a-92bf8d5f4132 None None] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:24.282207 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0108ea5-13ec-46be-894a-92bf8d5f4132 None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:24.282453 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0108ea5-13ec-46be-894a-92bf8d5f4132 None None] PATH_INFO: `/` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:24.285543 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1459/2933] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:51:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:51:24.295239 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9927075d-30c2-4c26-b9d2-b35f2875a283 None None] REQUEST_METHOD: `POST` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:24.296104 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9927075d-30c2-4c26-b9d2-b35f2875a283 None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:24.296471 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9927075d-30c2-4c26-b9d2-b35f2875a283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:24.356924 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.auth.core [None req-9927075d-30c2-4c26-b9d2-b35f2875a283 None None] MFA Rules not processed for user `8a22fb90c01a4c689550c0efc1fbaf71`. Rule list: `[]` (Enabled: `True`). {{(pid=80178) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:51:24.358282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9927075d-30c2-4c26-b9d2-b35f2875a283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:24.366311 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1475/2934] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:51:24 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 07 01:51:24.384408 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0108ea5-13ec-46be-894a-92bf8d5f4132 None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:51:24.387107 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-d0108ea5-13ec-46be-894a-92bf8d5f4132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:24.411648 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0108ea5-13ec-46be-894a-92bf8d5f4132 None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:51:24.417972 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d96a3c0-7e77-4bd9-87a3-cef104317510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:51:24.421486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d96a3c0-7e77-4bd9-87a3-cef104317510 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:24.421486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d96a3c0-7e77-4bd9-87a3-cef104317510 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:24.421486 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d96a3c0-7e77-4bd9-87a3-cef104317510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:24.435119 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d96a3c0-7e77-4bd9-87a3-cef104317510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:51:24.438044 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d96a3c0-7e77-4bd9-87a3-cef104317510 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:51:24.454180 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1460/2935] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:51:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:51:24.552211 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56106263-b5ed-4814-94a8-033bbc51bdac None None] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:24.552794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56106263-b5ed-4814-94a8-033bbc51bdac None None] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:24.553282 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56106263-b5ed-4814-94a8-033bbc51bdac None None] PATH_INFO: `/` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:24.555797 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1476/2936] 10.222.0.26 () {54 vars in 921 bytes} [Sun Aug 7 01:51:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 07 01:51:24.562774 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] REQUEST_METHOD: `POST` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:24.563060 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:24.563339 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:24.612651 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.auth.core [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] MFA Rules not processed for user `9f6e6a410517438885907f00fabe80f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80179) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 07 01:51:24.613919 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:24.624198 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1461/2937] 10.222.0.26 () {58 vars in 1018 bytes} [Sun Aug 7 01:51:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 07 01:51:24.636486 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56106263-b5ed-4814-94a8-033bbc51bdac None None] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:51:24.639142 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-56106263-b5ed-4814-94a8-033bbc51bdac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:24.655666 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56106263-b5ed-4814-94a8-033bbc51bdac None None] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:51:24.659579 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:51:24.660216 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:24.660535 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:24.660813 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:24.673349 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:51:24.675064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:51:24.688177 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1477/2938] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:51:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:51:26.818859 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:51:26.831840 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77048df9-2d04-4737-82b4-c224ba05b24b None None] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:51:26.836226 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:51:26.837069 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:51:26.837409 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:51:26.837737 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:51:26.839906 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:51:26.858941 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:51:26.860692 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:51:26.878512 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1462/2939] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:51:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:51:34.682024 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:51:36.869509 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=21 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:52:57.463708 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:52:57.466027 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:52:57.485794 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a2e755-96ab-4846-8268-529768978f7f service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:52:57.493115 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9f6e6a410517438885907f00fabe80f0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:52:57.494637 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:52:57.494925 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:52:57.495205 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:52:57.497494 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:52:57.511707 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:52:57.513382 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:52:57.526528 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1478/2940] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:52:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:53:03.614597 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:03.616871 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:03.739506 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3262cb22-c132-4a0d-bd42-1615302b5325 service neutron] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:03.745548 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:03.746295 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:03.746659 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:03.747004 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:03.749152 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:03.763817 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:03.765903 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:03.783143 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1463/2941] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:53:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:53:05.499604 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:05.523019 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe7a477-824b-49da-ac38-8b2440c45a19 service nova] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:05.526975 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8a22fb90c01a4c689550c0efc1fbaf71', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:05.527588 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:05.527865 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:05.528135 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:05.530059 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:05.641888 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:05.644641 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:05.664689 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1479/2942] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:53:07.708617 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:07.728182 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86990ee6-2295-4f94-9568-55462b92a68b service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:07.737169 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:07.738230 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] REQUEST_METHOD: `GET` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:07.738723 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:07.739222 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:07.742400 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:07.761025 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:07.764220 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:07.786037 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:07.789062 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:07.790857 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1464/2943] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:53:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 07 01:53:07.821172 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b790ef8-dd83-4040-a185-c61ccee66730 service neutron] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:07.828841 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:07.829937 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:07.830415 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:07.830879 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] PATH_INFO: `/v3/users/3059ad20d01b49adbb6dbb8ca75e451c` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:07.833462 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3059ad20d01b49adbb6dbb8ca75e451c)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:07.837152 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:07.899291 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3059ad20d01b49adbb6dbb8ca75e451c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:07.900158 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3059ad20d01b49adbb6dbb8ca75e451c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:07.900869 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3059ad20d01b49adbb6dbb8ca75e451c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:07.910425 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3059ad20d01b49adbb6dbb8ca75e451c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:07.919271 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3059ad20d01b49adbb6dbb8ca75e451c'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:07.939563 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1480/2944] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:53:07 2022] DELETE /identity/v3/users/3059ad20d01b49adbb6dbb8ca75e451c => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:53:08.154796 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:08.173740 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729010ea-e74b-46ed-9667-45ea54b027ed service placement] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:08.246516 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:08.247160 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:08.247444 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:08.247823 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] PATH_INFO: `/v3/projects/a3df9bc7e5f54e938a0ab2bdc841ecd3` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:08.249240 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3df9bc7e5f54e938a0ab2bdc841ecd3)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:08.251241 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:08.315750 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3df9bc7e5f54e938a0ab2bdc841ecd3', 'request_id': 'req-2608aadf-d5bf-4570-bd2c-e4a92267804d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.323301 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3df9bc7e5f54e938a0ab2bdc841ecd3', 'request_id': 'req-2608aadf-d5bf-4570-bd2c-e4a92267804d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.332883 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3df9bc7e5f54e938a0ab2bdc841ecd3', 'request_id': 'req-2608aadf-d5bf-4570-bd2c-e4a92267804d'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.348045 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] The token cache is being invalidate because project a3df9bc7e5f54e938a0ab2bdc841ecd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:53:08.348357 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.349421 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.351089 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1465/2945] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:53:08 2022] DELETE /identity/v3/projects/a3df9bc7e5f54e938a0ab2bdc841ecd3 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 07 01:53:08.355692 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:08.357466 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:08.372082 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b3fe67-9708-4f44-94db-628dfba9866f admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:08.424493 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:53:08.444682 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:08.445441 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:08.445843 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:08.446253 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] PATH_INFO: `/v3/users/08d7b062eb5a4c978c1c28d79a39ba1b` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:08.447624 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=08d7b062eb5a4c978c1c28d79a39ba1b)` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:08.449938 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:08.509611 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08d7b062eb5a4c978c1c28d79a39ba1b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.510514 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08d7b062eb5a4c978c1c28d79a39ba1b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.511391 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08d7b062eb5a4c978c1c28d79a39ba1b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.521307 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08d7b062eb5a4c978c1c28d79a39ba1b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.notifications [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08d7b062eb5a4c978c1c28d79a39ba1b'} {{(pid=80178) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.557351 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1481/2946] 10.222.0.26 () {58 vars in 1224 bytes} [Sun Aug 7 01:53:08 2022] DELETE /identity/v3/users/08d7b062eb5a4c978c1c28d79a39ba1b => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:53:08.770109 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Authenticating user token {{(pid=80179) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:53:08.771971 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.fernet_utils [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80179) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:53:08.790333 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2608aadf-d5bf-4570-bd2c-e4a92267804d admin admin] Validating token access rules against request {{(pid=80179) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:53:08.795144 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:53:08.878254 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a75da33429f640299cbc9644c9a49291', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3747bdb31f45b9bc97b22fb3bfe021', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80179) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:53:08.879564 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:53:08.879844 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:53:08.880116 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] PATH_INFO: `/v3/projects/643d5812093148458f0b699a07a15805` {{(pid=80179) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:53:08.881387 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=643d5812093148458f0b699a07a15805)` {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:53:08.883334 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] RBAC: Authorization granted {{(pid=80179) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:53:08.938701 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '643d5812093148458f0b699a07a15805', 'request_id': 'req-d6ebe675-be32-4051-84b7-e61359e97b02'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.945173 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '643d5812093148458f0b699a07a15805', 'request_id': 'req-d6ebe675-be32-4051-84b7-e61359e97b02'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.954049 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '643d5812093148458f0b699a07a15805', 'request_id': 'req-d6ebe675-be32-4051-84b7-e61359e97b02'} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.968815 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] The token cache is being invalidate because project 643d5812093148458f0b699a07a15805 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80179) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 07 01:53:08.969190 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.969995 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG keystone.notifications [None req-d6ebe675-be32-4051-84b7-e61359e97b02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80179) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 07 01:53:08.971727 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: [pid: 80179|app: 0|req: 1466/2947] 10.222.0.26 () {58 vars in 1233 bytes} [Sun Aug 7 01:53:08 2022] DELETE /identity/v3/projects/643d5812093148458f0b699a07a15805 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 07 01:53:18.553426 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=33,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:53:18.964599 dv-cinder-master-3-iscsi devstack@keystone.service[80179]: DEBUG dbcounter [-] [80179] Writing DB stats keystone:SELECT=48,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80179) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 07 01:54:40.103407 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Authenticating user token {{(pid=80178) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 07 01:54:40.105832 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:54:40.192537 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd46cef-8e26-42b4-8529-e720d6a9ab59 admin admin] Validating token access rules against request {{(pid=80178) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 07 01:54:40.271056 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '06551751330a406f8f0003153245b6a9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2f39278bb4be4af68b716f8d549c5155', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80178) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 07 01:54:40.271675 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] REQUEST_METHOD: `GET` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 07 01:54:40.271965 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] SCRIPT_NAME: `/identity` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 07 01:54:40.272244 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80178) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 07 01:54:40.274064 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.fernet_utils [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80178) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 07 01:54:40.288508 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 07 01:54:40.290306 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983cc5fb-6d98-4617-a410-a52236662a01 service placement] RBAC: Authorization granted {{(pid=80178) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 07 01:54:40.305569 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: [pid: 80178|app: 0|req: 1482/2948] 10.222.0.26 () {60 vars in 1365 bytes} [Sun Aug 7 01:54:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 07 01:54:50.299055 dv-cinder-master-3-iscsi devstack@keystone.service[80178]: DEBUG dbcounter [-] [80178] Writing DB stats keystone:SELECT=58 {{(pid=80178) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}